Tuesday, February 26, 2008

Twining: Iron Python DSL for DB Update III

{

My thoughts as exhibited in a little Python script now have a name: Twining. 

Unplanned illness slowed me down over the weekend, but the beauty of lambda expressions make transformations a breeze. I'll let the code document what I added.

cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
mytrans = {"Field1":"upper","Field1": lambda x: x.upper(),"Field3": lambda x: x[::-1]}
database(cn).table("Photos").transform(mytrans).copyto.csv("c:\\foo.csv")

As you can see, "mytrans" is a dictionary containing a matching of column names to both a keyword string, "upper", and also lambda expressions that model functions to operate on the constituent data (the [::-1] idiom reverses a string).  I'd add a few keywords for simple things people often do to data but the beauty of lambda expressions is that it keeps my code lazy and empowers whoever is using the script to make their own judgement about what to do with data.


Here's what goes on behind the scenes - the copyto class has a checkAndTransform method that matches up a field value with it's "transform":


	def checkAndTransform(self, field, value):
if registrar.transforms.has_key(field):
if repr(type(registrar.transforms[field])) == "<type 'function'>":
value = registrar.transforms[field](value)
if registrar.transforms[field] == "upper":
value = value.upper()
return value


I'm not Pythonic enough yet to know if I need to call repr and do a string comparison when testing if a type is a function, but it works just fine.


So a few thoughts:


1. What are the usual changes we make to data when exporting or importing?
2. Is it stupid to have shortcut strings or should I just fall back on writing lambdas?
3. Is there a better way of testing for a function than
repr(type(registrar.transforms[field])) == "<type 'function'>"


The to-do list from my previous post is still intact, I'll be working through it in the near future.  Have a look at the complete script and if you have thoughts or opinions, please let me know. A big goal here is not only to make something useful, but to learn Python (and be Pythonic, the two don't necessarily coincide).


Here's a full list of what can be done at this point:


mytrans = {"Field1":"upper","Field2": lambda x: x.upper(),"Field3": lambda x: x[::-1]}

database(cn).table("MyTable").transform(mytrans).copyto.csv("c:\\foo.csv") #with transform
database(cn).table("MyTable").copyto.csv("c:\\foo.csv") #without transform
database(cn).table("MyTable").copyto.database(cn2)
database(cn).table("MyTable").copyto.xmlFieldsAsAttributes("c:\\here.xml")
database(cn).table("MyTable").copyto.xmlFieldsAsElements("c:\\there.xml")
database(cn).backupto("C:\\temp\\my_backup.bak")



}

Thursday, February 21, 2008

Iron Python DSL for DB Update II

{

Attepting releasing early & often - here are a few new features to the ideas I'm putting together for the Database specific DSL.  I'll summarize all that's possible with it now with the new stuff highlighted in red.

#some connection strings
cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
cn2 = "Data Source=.\\sqlexpress;Initial Catalog=Irvine;Integrated Security=SSPI;"

#some things to do
database(cn).table("MyTable").copyto.csv("c:\\foo.csv")
database(cn).table("MyTable").copyto.database(cn2)
database(cn).table("MyTable").copyto.xmlFieldsAsAttributes("c:\\spam\\eggs.xml")
database(cn).table("MyTable").copyto.xmlFieldsAsElements("c:\\spam\\eggs.xml")
database(cn).backupto("C:\\spam\\eggs.bak")

The overall idea is to build a language centric, developer oriented implementation of what is done in a tool like SSIS or its predecessor DTS - I'm using SQL Server for a start but there's no reason this can't include other database platforms. It's quite exciting that even without knowing much Python I can get that much done in the language.  I'll spend time refactoring and reorganizing the script I've started with, but please feel free to download and modify it. Please post any improvements or ideas for how I can improve the Python and added any other features to the dsl. (Point on using __getattr__ well taken).

My todo list?
1. Aforementioned "transforms" for data in migration
2. Exception handling :^)
3. Copying tables is a rough sketch but needs a lot of work
4. Export formats besides CSV and XML: arbitrary delimiters and fixed field length
5. __getattr__ :^)

}

Tuesday, February 19, 2008

Iron Python DSL for ETL

{

Hanselman wrote about "fluent" interfaces recently which confirmed something I'd been thinking for a while. What if rather than massive tools we just had a language to express ourselves when we wanted to move data around?  What if we could say something like:

database("connection string").table("table").copyto.csv("C:\\spam\\eggs.csv")
or
database("connetion string").table("table").copyto.database("connection string 2")

In the spirit of releasing early and often, here's an implementation of the above language for sql server with Python. I'll be extending the flexibility quite a bit, and my Python skills are still those of a learner, but here it is for comment.  Before I post the whole script, here are a few goals:
1. XML data:
database("connection string").table("table").copyto.xml("C:\\spam\\eggs.xml")
2. Transforms:
transformDictionary = {'column1':'upper', 'column2':'some regex'} #these could become a fluent interface of their own.
database("connection string').table("table").withTransform(transformDictionary).copyto.csv("C:\\spam\\eggs.csv")

So here's that rather unrefined code download the latest implementation

Below was just some scratchwork from a Friday afternoon when I started sketching:

# BORRE DATABASE OBJECT
# 02/08/2008 - David Seruyange - @FRIDAY BURNOUT
import clr
clr.AddReference("System.Data")
from System import *
from System.Data import *
from System.Data.SqlClient import *
from System.IO import *

class registrar:
tableRegister = ""
databaseRegister = ""

class database:
"Model of a database somewhere"
def __init__(self, connectString):
"Initialize the rectangle"
self.connect = connectString
registrar.databaseRegister = connectString
self.copyto = copyto()

def table(self, t):
registrar.tableRegister = t
return self

class copyto:
"Model of a copy of some object"
def csv(self, path):
""" Save the result to a CSV file """
cn = SqlConnection(registrar.databaseRegister)
cn.Open()
cmd = SqlCommand("select * from " + registrar.tableRegister, cn)
r = cmd.ExecuteReader()
fileOut = open(path, 'w')
while r.Read():
sOut = ""
for i in range(r.FieldCount):
sOut += str(r[i]) + ","
sOut += "\n"
fileOut.write(sOut)
cn.Close()
fileOut.close()

def database(self, connect):
(ddl, colInfo) = buildDDL(registrar.tableRegister)
cnTarget = SqlConnection(connect)
cnTarget.Open()
cmdBuildTargetTable = SqlCommand(ddl, cnTarget)
cmdBuildTargetTable.ExecuteNonQuery() #builds the target table
cnSource = SqlConnection(registrar.databaseRegister)
cnSource.Open()
cmdSourceIter = SqlCommand("select * from " + registrar.tableRegister, cnSource)
sourceReader = cmdSourceIter.ExecuteReader()
insertCode = ""
while(sourceReader.Read()):
insertTemplate = "INSERT INTO " + registrar.tableRegister + " VALUES("
for i in range(sourceReader.FieldCount):
insertTemplate += "'" + str(sourceReader[i]) + "',"
insertCode += insertTemplate[:-1] + ")\n"
cmdExecuteInsert = SqlCommand(insertCode, cnTarget)
cmdExecuteInsert.ExecuteNonQuery()
# "save: %s\n table: %s\n to path: %s" % (registrar.databaseRegister, registrar.tableRegister, connect)

def buildDDL(tName):
query = """
select
c.[name], c.max_length, c.[precision], c.scale,
ty.[name] as typename
from
sys.columns c
join sys.tables t on c.object_id = t.object_id
join sys.types ty on c.system_type_id = ty.system_type_id
where
t.[name] = '__table__'
"""
query = query.replace('__table__', tName)
cn = SqlConnection(registrar.databaseRegister)
cn.Open()
cmd = SqlCommand(query, cn)
r = cmd.ExecuteReader()

ddl = """
CREATE TABLE __xyz__(
__columns__)
"""

columnsDefinition = ""
colTypeInfo = []
while r.Read():
#todo: support for other column types such as numeric or decimal
columnTemplate = " COLUMN_NAME DATATYPE LENGTH NULL,"
columnTemplate = columnTemplate.replace('COLUMN_NAME', str(r[0]))
columnTemplate = columnTemplate.replace('DATATYPE', str(r[4]))
colTypeInfo.append(str(r[4]))
if str(r[4]) == "varchar":
columnTemplate = columnTemplate.replace('LENGTH', "(" + str(r[1]) + ")")
elif str(r[4]) == "char":
columnTemplate = columnTemplate.replace('LENGTH', "(" + str(r[1]) + ")")
else:
columnTemplate = columnTemplate.replace('LENGTH', "")
columnsDefinition += columnTemplate + "\n"

r.Close()
cn.Close()
ddl = ddl.replace('__xyz__', tName).replace('__columns__', columnsDefinition).replace(",\n)", "\n\t)")
return [ddl, colTypeInfo]


cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
cn2 = "Data Source=.\\sqlexpress;Initial Catalog=Irvine;Integrated Security=SSPI;"
# jepp
#database(cn).table("Photos").copyto.csv("c:\\foo.csv")
#print "---"
database(cn).table("Photos").copyto.database(cn2)

}

Monday, February 18, 2008

Release Early & Often

{

Paul Buchheit (who we can thank for Gmail) comments on a post by Marc Andreessen:

For web based products at least, there's another very powerful technique: release early and iterate. The sooner you can start testing your ideas, the sooner you can start fixing them.

I wrote the first version of Gmail in one day. It was not very impressive. All I did was stuff my own email into the Google Groups (Usenet) indexing engine. I sent it out to a few people for feedback, and they said that it was somewhat useful, but it would be better if it searched over their email instead of mine. That was version two. After I released that people started wanting the ability to respond to email as well. That was version three. That process went on for a couple of years inside of Google before we released to the world.

Interesting and it goes along with something Guy Kawasaki was saying in a recent podcast I commented on hearing.  So the next question comes down to technique - when you're working on something by yourself, how do you schedule builds?  I'm thinking I may start to take deliberate "coding vacations" - days I take off* from work in order to push through releases on my various little software projects.  Hm... other thoughts?

*My employer offers "comp days" so it's not at the expense of my day job.

}

Saturday, February 16, 2008

Check File Existence, Javascript (Prototype)

{

If you're like me, you spend a lot of time using System.IO and can easily check for the existence of a file on the filesystem with a File.Exists(@"C:\spam\eggs.file")

It's a little surprising that I hadn't run into it before but a few weeks ago I needed that same functionality with javascript. I found a few things online and what works best is using AJAX to make a request and check the HTTP status for whether or not the request failed.  My approach was to use Prototype and something like this:

function checkForFile(imgName, displayHolder){
new Ajax.Request('Repo/' + imgName + '.jpg',
{
method:'get',
onSuccess: function(transport){
var response = transport.status || "no response text";
if(parseInt(response) == 200){
displayHolder.src = 'Repo/' + imgName + '.jpg';
displayHolder.style.display='block';
}
},
onFailure: function(){
displayHolder.src = 'Repo/NoImage.gif';
displayHolder.style.display='block';
}
});
}

Works like a charm.


}

Mantras

{

Some good things to repeat to yourself from Dormando. I've a long way to go on so many of these and was geeky enough to give myself a scorecard.

I'll spare you that (after just deleting a lot of babble that would only be interesting to me) but there are two areas that I can do with some serious improvement:

1. Relentless automation
It's shameful how often I'm doing things I could easily automate (repeated FTP sessions?)
2. Monitoring
I wonder how out of line it would be to programmatically enable "breadcrumbs" for user's activity?  Finding out what they really used and what they insisted upon for nothing.

}

Saturday, February 09, 2008

Spitting Code Like Lam

{

I started changing my Visual Studio settings after seeing Hanselman's post and thought I'd be able to spit fire like John Lam once I'd downloaded his settings (it's gotta be the shoes!). The contrast was a bit much (though working in low light helped) and after a night's worth of being like JLam I wanted to go back to (dare I admit?) VS default settings. That didn't work out too well (even using the "default settings" option in VS, go figure) so I opted for Brad Wilson's settings which I've grown to like.

When the IDE "hot or not" thing first cropped up I changed settings but what makes me boring, preferring the default, is that I'm on a lot of machines at different times; my laptop, desktop, remote on a couple of servers, and when I help others.

}

Monday, February 04, 2008

Grabbing SQL 2005 Blobs with IronPython

{

On Saturday I needed to grab some blob data and create files out of it. The following was some Python I used, inspired very much by this code.

The biggest point of interest is the strongly typed array, which in IronPython is simple once you know how: using Array.CreateInstance.

 

 

#GET IMAGES FROM IMAGE BINARY FIELD
# 02/02/2008 - David - @THE COMMAND CENTRE

import clr
clr.AddReference("System.Data")
from System import *
from System.Data import *
from System.Data.SqlClient import *
from System.IO import *

CONNECT = "Data Source=(local)\sqlexpress;Initial Catalog=DATABASE;Integrated Security=SSPI"

sql = "select ID_FIELD, BLOB_FIELD from product_set"
cn = SqlConnection(CONNECT)
cn.Open()
cmd = SqlCommand(sql, cn)
reader = cmd.ExecuteReader(CommandBehavior.CloseConnection)
buffSize = 100
buff = Array.CreateInstance(Byte, buffSize)
recCounter = 0
while reader.Read():
print "Creating " + str(reader[0])
fs = FileStream(str(reader[0]) + str(recCounter) + "_diagram.png", FileMode.OpenOrCreate, FileAccess.Write)
bw = BinaryWriter(fs)
if(reader[1] != DBNull.Value):
posit = 0
b = reader.GetBytes(1, posit, buff, 0, buffSize)
while(b == buffSize):
bw.Write(buff)
bw.Flush()
posit = posit + buffSize
b = reader.GetBytes(1, posit, buff, 0, buffSize)
bw.Write(buff, 0, b)
bw.Flush()
bw.Close()
fs.Close()
recCounter = recCounter + 1

reader.Close()


I'd like to clean it up a bit, but I've written about 5,000,000 things that I "intend" to clean up before posting and they're never posted.


}

What Comes First; Rocks

{

Oliver Steele has a new essay entitled Adding the Easy Piece; or, The Metaphor of the Rock.  Insightful as usual, and I love the visualizations.

Personally, I'm experiencing the effects of pushing a rather large rock around a room.

}

Wednesday, January 30, 2008

Scott McCloud's Four Tribes - Developer Edition

{

Last year I was fortunate to catch Scott McCloud here in Sioux Falls. It's doubtful that he'd ever find his way here under normal circumstances but he was doing a 50 state tour in support of his book, "Making Comics."  I believe a dual aim was to provide his two young daughters, who travelled and presented with him, an experience of a lifetime. 

I was particularly struck with a classification he had for comic (and indeed all types) artists.  He placed them into four quadrants:

Here is a quote of an explanation of how the classifications work:

Those who value draftsmanship the most, and who are the most invested in the idea of mastery, beauty, and craft find themselves on the opposite side of the fence from those whose strongest values are raw honesty, authenticity, and a kind of rebellion against the status quo. That dichotomy comes between what I call the classicists and the iconoclasts. The children of Hal Foster versus the children of R. Crumb.

That’s the truth and beauty diagonal that separates two of the four tribes. The other two corners are the formalists and the animists, who represent the distinction between form and content. The formalists like to experiment and take the medium apart, figure out how it works and put it back together, and try new things; while the animists really just like to tell a compelling story, and want the form to vanish, to become transparent, so that you don’t know you’re reading a comic at all.

I wonder how well these notions apply to the practice of programming.  I can easily see the classicist: mastery, beauty, and craft - the person who uses things like recursion excessively for the computer science elegance they bestow.  Opposite them, I see the iconoclast -  the hacks that make things work and could care less about the principles or structure of a thing and instead devote themselves to the beauty of experience.

I think the animists and formalists can be distinguished as well by the metric of language/tool preference.  The obsessives of language would seem to tend toward formalism and understanding every moving part.  An animist may be the type of person who excels at making a tool do legwork with no particular interest in exactly what that tool is baking under the covers. 

Of course we all will have elements of each but I like to think of myself patterned after Scott McCloud. The formalist who might write the same software over and over in different languages just to see how expression and power vary at the lowest levels.

Interesting food for thought.

}

Tuesday, January 29, 2008

Java Programmers on Windows Forms team?

{

Randomly: worked with the CheckedListBox tonight. It's interesting to run into "getter/setter" type methods in the .NET universe.  One would assume a collection that specifically supported setting constituent objects as "selected" or "checked" but instead there are methods like: GetItemCheckedState and SetItemCheckedState.

It would be interesting to have the Windows Forms equivalent to the annotated base class library reference. Perhaps tomorrow I'll try to peak at the source code.

}

Disturbing News

{

"I am going to tell you something that will disturb you. You might laugh, but it will be a cold uncertain laugh that will haunt you as you read on, because somewhere deep down you'll know it to be true. You might brush it off, get on with your day, yet sometime later, a week or a year, it will seep back in and unsettle you to the core. From that moment on you will be changed. You will think different, act different and will fundamentally be different. So take a moment to prepare yourself now, breath deeply, clear your mind and open up to the possibility that building software is hard."

Interesting thoughts from The Wayward Weblog

 

}

Thursday, January 24, 2008

Not An Engineer

{

Ravi Mohan makes the provocative, if not new claim (with follow up) that programmers are not engineers. I take the majority of his quibble to be the title some flaunt: "software engineer."

I've never felt able to use "engineer" for myself, but I don't perceive this badly. My official title is "software architect" though most of my day is not spent on architecture.

It's a bit postmodern of me but does anyone's title mean much?  Short of very specialized work, one must have multiple skillsets for the different roles they are cast in on a daily basis.

I have to do the following, fairly regularly:
1. Write Code
2. Write Specs
3. Customer Service
4. Testing
5. Write Proposals
6. Mentor
7. Marketing

I'm owning up to a lot of stuff, particulary #7, that I would never admit to for the sake of saying that most jobs these days require multiple intelligences.

Think Santiago Calatrava: engineer, architect, artist. And he deals with clients, too.

}

Temporary Switch Back: C# to VB.NET

{

I did the switch back for a project that a VBA enlightened client requested but I kept finding myself asking how to syntactically do what I usually do in C# (especially inheritance-wise) in VB.NET.  Terry Smith has a handy online book that has a comparison so that you can easily figure out what the VB.NET syntax is for what you know how to do in C#. I'm not sure when it was written; no word on Generics but that's an easy enough syntax to duplicate with (Of T) rather than angle brackets.  What throws me off in addition to the inheritance verbosity are a lot of the array related casting operations that seem intuitive now in C# like:

Dim dropRows As Row() = e.Data.GetData(GetType(Row()))

Of course with the latest developments in VB.NET, I may end up using the language enough to make all of this easy to recall from memory. 

As much as I want to join the Red Sox fans in cheering for VB, there are still many things that annoy me about the language.  Line continuation characters are at the top of the list, and case insensitivity closely following. Yes, I strongly dislike case insensitivity in a programming language.

}

Tuesday, January 22, 2008

Up In The (Ruby) Club

{

A while back I had a phone call from a woman with an eastern European accent wondering if I would be willing to take a survey.  I'm usually an instant no to queries like this but she knew my Achilles heel:

"Participants get $50 Amazon Gift Certificate"

So a few days ago my inbox chimed with an email with the gift certificate in tow. I didn't hesitate with my purchase of David Flanagan's Ruby book

As an aside, it seems that Flanagan got some flack from a Gilles Bowkett on not being a bona fide Ruby person. If Flanagan isn't cultured enough to do Ruby, I'm in trouble...

... but I've been in trouble most of my short puff anyway.

On a serious note: the "up in the club" attitude of some (okay, mostly Rails) Ruby people is what is driving a backlash against Rails and Ruby in general.

}

Friday, January 18, 2008

Nutrition

{

I just looked at Oliver Steele's programmer's food pyramid and did a self evaluation of sorts. I know that I err too much on the reading of "blogs" versus reading long prose about programming.  That's something to change this year.

As well I use libraries a lot without digging into anything once my problem solving is done.  I think there's a form of mental stamina in digging deeper into a problem once you've apparently solved it - my instinct is to simply move on and try to tackle what's next.  Now that the .NET Framework code is available, that and other libraries should occupy my reading time.

While I don't struggle with the inclination to write code, I've long since thought it's better to try to write different types of applications and build some breadth.  Especially after my umpteenth web based CRUD app, I'm now interested in blending Windows Forms and Web stuff in the small variety of utilities that I compulsively write.

}

Wednesday, January 16, 2008

What's It Worth?

{

In the wake of all the plastic surgery and body augmentation going on around, a developer like myself can feel a little out of place - out of century to be more exact. 

And yet, whether he's joking or not, I'm half tempted by Wesner's admission to "performance enhancers."

But the thing I know with certainty is that being productive will only beget the desire to be more productive. Instead of learning and implementing 4 technologies, it will be 8.  I'll pile more home projects and "meware" until I'm aching for another 5 or 6 hours a day.

In a world that has the audacity to produce ridiculously smart people who are subject to feeling that yet others are "smarter" I may as well hold.

But it's tempting... maybe until I know F# and then I'll hold...

}

Saturday, January 12, 2008

13.75 hours, CodeMash

{

I earned the reputation as "the guy from South Dakota" at CodeMash; many people were amazed that I took it upon myself to drive just under 14 hours both there and back for the conference.

What would they have done if I told them I also:
a) took time off from work
b) paid for it out of pocket
c) drove my own car

But I think it's not *that* strange. Staying at work would have meant the same old stuff I do every day all year and instead I got to see some of the best Einstein's presenting on Python, DSLs, Dojo, REST, and IIS7. I made some friends as well and on top of that got to kick it with Scott Hanselman.  All for the cost of a long drive with an audio book.

I had posted on sessions I wanted to see, here's a little rundown of memorable sessions I actually attended:

1. Neal Ford's keynote "Software Engineering & Polyglot Programming"

Neal's one of those sedate cognoscenti who don't appear to have an emotional angle on platforms, just a sense of design and engineering that they wish to communicate.  In his talk he urged developers towards the notion of language as a tool, and that dynamic languages take us places that our old friends in the static realm cannot.  He made a reference to Jack Reeve's essay which I've written about before, and said a few times the following phrase which I thought was apt:
"Testing is the engineering rigor of software development."

If I could have asked Neal, I'd have wanted some comment on something I posted some time back - Oliver Steele's blog post on The IDE Divide in which he divides developers into the Language Mavens and Tool Mavens.  I think Neal's perspective was very much that of a language versus tool maven.

2. Catherine Devlin's "Crash, Smash, Kaboom Course in Python"
Great talk on Python fundamentals while illustrating a universe simulator library (forget the name).  I've been trying to pick up Python for some time and much of what I got was from watching Catherine's use of the language in her demos.  The one bummer: I was using a Python distribution from my iBook (Mac OSX) that didn't include the Python modules she was demonstrating with. Even more of a bummer: in my hotel room a different laptop was running Ubuntu with just the modules needed to follow along with her in code.

3. Kevin Dangoor on Dojo
I may have made Kevin feel uncomfortable with how happy I was to see this talk (he did a great job) but I've felt quite alone over the last two-ish years working with javascript libraries.  I used jQuery heavily for my tool "proper" after which I used Prototype/Scriptaculous for a few other projects.  I then switched to YUI for nRegex so I'd been around the block with my frameworks with only Dojo missing.  Dojo seems to go beyond these other frameworks just on its ambition alone and I'm wondering if that can translate into a happy mix with the heavyweight stuff I have to do at work with ASP.NET. 

4. Dustin Campbell on F#
Dustin gave a deft introduction to the F# language which had to be the biggest thoughtbludgeon I got during the conference.  I've listed F# as my "someday language" and it will remain so since Ruby will occupy my time along with Python in learning mode over the next year or so. Nevertheless, it was good to see a demonstration of the language and some of the implications thereof.

5. Kevin Dangoor on Gears and Dojo (DojoX Offline)
The second session I sat in on with Kevin, this dealing with Dojo's offline capabilities in large part courtesy of Google Gears.  It was as good as the first Dojo session and left my mind popping with ideas for leveraging that functionality. 
Soon - I'd say around the release of Silverlight 2.0 - people like me are going to face a fork in the road of using browser plugins and thick (they use the word "rich") vendor technologies, or using javascript and existing web technologies like CSS on a new level.  All the Dojo stuff represents, to me, what Microsoft and Adobe (no need to even mention JavaFX, oops I just did) are not tuning to as they ramp up Silverlight and Flex.  Not that it's in their interests: the stuff of web standards isn't really the profit center that proprietary technologies are.

6. Neal Ford on DSLs with Dynamic Languages
Neal was impressive this second time with some practical talk and examples of DSLs.  He started first with Groovy, which left me disappointed since I don't anticipate using it, but to show the "best" world he switched to Ruby.  I am really looking forward to IronRuby's release, but at present I think much of what he discussed can be implemented in Python. 

7. Dave Donaldson on RESTful Web Services
I've been familiar with the REST buzzword for some time but as a term thrown out in competition with standard web services (the WS-* stuff).  Dave did a good job of unpacking the essence of REST and showing demonstrations of usage.  It's interesting to see just how much backlash there is against WS-* and the manufactured complexity therein.  The only thing I disagreed with was Dave belittling Dave expressed reservations about the Astoria project teetering into the over-engineered realm of WS-* but I am hopeful it will retain its original intent - I really, really like the idea of Astoria (born of the same backlash to WS-*? I'm just talking here...).

8. Scott Hanselman on Mashups with IIS7
He has the gift and he knows it: first with a humorous prologue and then with a good presentation on IIS7 and using its underpinnings with a mix of PHP. Inspired by Bill Staples article on IIS7 and PHP, but with the detail and demonstrations with which Scott's earned his following.

I went to a few other sessions too, but these were the ones which, as the Brits say, "tickled my fancy."  I was disappointed to miss out on Jay Wren's intro to Castle as well as Bruce Eckel's thoughts on Python. I guess there's next year...

... and I do anticipate returning to CodeMash although I'm quite sure I'll fly rather than driving. :^)

}

Wednesday, January 02, 2008

Enemies and Organization

{

I usually try not to be an echo chamber, especially when posts show up on Coding Horror or Steve Yegge's blog. But what's interesting is a rebuttle to a notion that Yegge started, namely that size is an enemy in code base.

Franz Bouma has an interesting post to follow up that is worth a counter balance of views.

Conventions:

I commented once about LoC in one of our projects, not so much with an interest in maintainability, but as a comparison with how much code was being "generated" on my behalf by the IDE.  The project has grown since that post which had the codebase at around 30,000 lines of code.  This is nowhere near the 500,000 line mark but one design decision made early on that has always helped in navigating around was the heavy use of conventions.  Because we followed a pattern for where we put things, it's easy to find which project a piece of code is in as well as which part of the code one needs to dive into to make a modification. 

Abstraction:

One thing I'm noticing about the open source projects I look at is how layers of abstraction can lead to the kind of indirection that's hard to follow.  I speak only of ideals but it seems like too much abstraction can lead to the kind of confusing codebases that people online are reacting to. That logic may be the Mort in me going against the grain; it seems like many think that indirection and abstraction may add flexibility, but that flexibility will be a trade off for how easily a person, especially not the original author, finds their way.

}

Sunday, December 30, 2007

xPathFinder

{

The short version:
I'm releasing xPathFinder, a tool I wrote for doing XPATH expressions against xml files on the file system. You can get information about it as a tool by going to the website. Here are some screenshots:

 

The long version:
Not too long ago I ran into a problem that I thought would be common but didn't seem to have a canned solution. We use a webservice to store some data in a database but we also store the xml files transmitted on the file system as a defensive maneuver in case the database is unavailable or the xml is invalid. After a while I was dealing with a lot of xml files - far too many to search one at a time and needing more structured lookup than a desktop search tool offered. This is the raison d'etre of this tool. Of course it could have been a quick script in -insert language of choice- but this was something I needed to do repeatedly and have more facility with than console output. I'm also planning to make it simple enough for our client to use. It was a goal of mine to get it up before the end of the year so I'm happily posting this a day early.

One more footnote to this tool - stylistically I've done quite a bit of blending between Windows Forms and Web by leveraging the WebBrowser control heavily for things I thought would be easier to quickly generate html for as well as the feedback mechanism which allows users to report bugs or request features. It's so tightly coupled that while it will work without an internet connection, it's a much more limited experience. I really like this model and will probably start doing more blending like this for new work.

You can get more detail from the website or just ask directly by submitting feedback.

Future plans?

1. XPATH functionality works for most xml documents, but namespace heavy documents with prefixes seem to throw a curve ball.
2. What I'm posting was actually a fast prototype I intended to use to make this tool in WPF from - I'm not sure that the WPF step is still necessary but I may release an updated version since this is a fairly straight forward utility.
3. I'd like to make it open source. Not sure of what steps to follow, but CodePlex is probably the best starting point. I always promise myself I'll clean up my code before doing something like that but maybe I should just bite the bullet since I think the idea is sound enough.

}