Wednesday, April 02, 2008

Generate Catalog Images in IronPython

{

What if you had a client who has a big product catalog you're integrating with the web.  You work out a convention for how data is displayed and want to generate some test images for yourself...

First, just some basics on generating a single image:

import clr
clr.AddReference("System.Drawing")
from System import *
from System.Drawing import *

def GenerateImage(text):
starter = Bitmap(1,1)
g = Graphics.FromImage(starter)
f = Font("Arial", 14)
theImage = Bitmap(starter, Size(200,200))
g = Graphics.FromImage(theImage)
g.Clear(Color.Orange)
g.DrawString(text, f, SolidBrush(Color.DarkBlue), 0, 0)
g.Flush()
theImage.Save("c:\\temp\\pyImage.gif")

if __name__ == "__main__":
GenerateImage("Holly was a hunter")



Here's some TSQL for the sample data I'll use:

create table testdata(
testid int identity(1,1),
testvalue varchar(50)
)

declare @i int
set @i = 0
while @i < 25 begin
insert into testdata(testvalue)
values('PRODUCT ' + convert(varchar(2), @i))
set @i = @i + 1
end



Now for a finalized version that hits the database and generates images:

import clr
clr.AddReference("System.Drawing")
clr.AddReference("System.Data")

from System import *
from System.Drawing import *
from System.Data import *
from System.Data.SqlClient import *

def GenerateImage(text, path):
starter = Bitmap(1,1)
g = Graphics.FromImage(starter)
f = Font("Arial", 14)
theImage = Bitmap(starter, Size(200,200))
g = Graphics.FromImage(theImage)
g.Clear(Color.Orange)
g.DrawString(text, f, SolidBrush(Color.DarkBlue), 0, 0)
g.Flush()
theImage.Save(path)

def GenerateImagesFromDatabase(connectString, query, sourceColumn, outPath):
cn = SqlConnection(connectString)
cn.Open()
cmd = SqlCommand(query, cn)
r = cmd.ExecuteReader(CommandBehavior.CloseConnection)
while r.Read():
baseText = r[sourceColumn].ToString()
GenerateImage(baseText, outPath + "\\" + baseText + ".gif")
r.Close()

if __name__ == "
__main__":
connect = "
Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
sql = "
select * from testdata"
col = "
testvalue"
GenerateImagesFromDatabase(connect, sql, col, "
c:\\temp\\fobot")




 


Pretty cool stuff, I hope it's useful to more people than just myself.  Download the first script here, the second more elaborate one here.


}

Tuesday, April 01, 2008

Twining

{

I've been quiet but still working on Twining.  Unfortunately lots of distractions at work have slowed me down. Most of what I've done of late is adding unit tests for the functionality that already exists.  I twittered that adding unit tests a posteriori is not fun and tends to have the effect of one simply verifying what they've done with pasted code.   Now that most of that is done I'd like to add a few things:

1. Operating on queries like tables for export and copy.  Everywhere that you see "table" below, I'd like to have "query":

database(cn).table("MyTable").transform(mytrans).copyto.database(cn2)
database(cn).table("MyTable").transform(mytrans).copyto.xmlFieldsAsAttributes("c:\\here.xml")
database(cn).table("MyTable").transform(mytrans).copyto.xmlFieldsAsElements("c:\\there.xml")
database(cn).table("MyTable").transform(mytrans).copyto.delimited("\t", "c:\\foaf.tsv")
database(cn).table("MyTable").copyto.delimited("\t", "c:\\foaf_no_trans.tsv")
database(cn).table("MyTable").transform(mytrans).copyto.csv("c:\\foo.csv")
database(cn).table("MyTable").copyto.xmlFieldsAsAttributes("c:\\here.xml")
database(cn).table("MyTable").copyto.xmlFieldsAsElements("c:\\there.xml")
how about: 
database(cn).query("select this, that, [the other] from foo").copyto.csv("c:\\querydata.csv")





2. Support for SqlBulkCopy - as far as I know this is only available between Sql Servers (or, more specifically SqlConnection instances).


3. Cloning of tables - if you use sql server, this is quite simple with the following TSQL:

SELECT * INTO MYCOPYTABLE FROM MYORIGINALTABLE WHERE 1 = 2



4. Database "ping" - just a simple way to test connection strings.

cn = "my connection string"
database(cn).ping()

5. Inspired by ConfigObj I'm wondering what a JSON like table definition would look like. In ConfigObj, you can define things like:

section2 = {
'keyword5': 'value5',
'keyword6': 'value6',
'sub-section': {
'keyword7': 'value7'
}
}

I wonder if one could have the following for a table:

tableDef = {
'personid': {
'datatype':'int',
'identity':'true',
'seed':1
},
'firstname':['varchar',50, 'null']
}


Just something to gnaw on...

 


I'll hopefully get a chance to work on this stuff soon.


}

Saturday, March 29, 2008

Data Access Strategy

{

What data access strategy do you use in your .NET work? The question has been on the mind of Scott Hanselman it seems because the last few podcasts have covered two different strategies - LINQ and CLSA.  Tonight I noticed that the ADO guy has a poll of his own with some interesting results:

Web Poll Powered By MicroPoll

 

I will admit that I'm pretty consistently using ADO.NET + Datasets but more so because the project I'm working on was well under way before the release of VS 2008. I wonder of those responding how many are working on new versus older, longer projects?

}

Wednesday, March 26, 2008

In Defense Of Lurkers

{

Hanselman's 7 Blogging Statistics Rules... post was good medicine for a haphazard blogger like me. There was one portion to which I had comment:

I feel like we've (that means me and you, Dear Reader) have a little community here. When you comment, I am happy because I feel more connected to the conversation as this blog is my 3rd place. I blog to be social, not to have a soapbox. I'm even happier when the comments are better and more substantive than the post itself. I would take half the traffic and twice the comments any day. If you're a "lurker," why not join the conversation?

I've spent most of my time on the web lurking.  It started with a mailing list for people who like electronic music ("IDM") I joined while in college - in the 10 years it's been I've probably posted less than 20 messages.  But I have spent a lot of time as a curator for posts I've enjoyed as well as making notes of what music I would investigate the next time I was in a record store. 

In a similar fashion I tend to hang onto posts I enjoy and go back to them.  To out myself further as a geek I even print out the ones I'd like to focus on so that I can read them offline at a coffee house or bookstore. I file them away and when I do my occasional cleaning it's hard not to smile and pause to re-read something that was good on the first effort. It's hard to measure us lurkers because we don't vocalize our responses right away and when we do finally get them the post may have tens or hundreds of comments at which point all seems moot.  But I think as a lurker I can be a better blog consumer in the sense that I hang onto what's said a little more - it's not just a random post I skimmed in the aggregator. 

Forgive the lurker as they make sputtering or failed attempts at immediate conversation, but maybe that conversation will be like the one you have with a great dead tree publication being read and reread.

}

simplejson

{

simplejson is a JSON library for Python 2.3+, courtesy of here.  A while back I goofed off with IronPython and Prototype writing a web based "shell" inerface.  I may have to return to that project although my problem wasn't really with encoding, it was with keeping context while using multiple commands (like cd .. and then dir with the new directory as your location) in a session with cmd.exe. I've been meaning to write the PoshConsole guy to ask specifically about this...

}

Chris Wilson interviewed on Pixel8

{

The interview is here, it's good stuff. Imagine a half billion users. That's a humbling thought...

}

Tuesday, March 25, 2008

Jeff Zeldman, Pixel8 Interview

{

Joel's got a big enough voice that I'm sure most people didn't miss his Martian Headphones essay which explained the futility of standards.  Jeff Zeldman, who I've always been aware of in my time lurking on the web chimes in on web standards on Craig Shoemaker's new gig, "Pixel 8."  The podcast sound is very rough but it's a good context builder for the notion of standards and what they mean.  If you've ever had to wrangle with a different look in different browsers, you appreciate standards even if they are complicated and never achieve perfection.

Just noticed that Pixel8 has a podcast with Chris Wilson. That will have to wait until tomorrow -

}

Unit Tests with Iron Python

{

"Testing is the engineering rigor of software development" - Neal Ford

Got a little tip from fuzzyman on using unittest for Unit Testing with Iron Python. Because I'm a slouch I had an old IronPython 1.1 release and kept getting a BaseException error when my tests failed.  The class has since been implemented and should work with the latest version of IronPython that you can download (I'm using a beta release of 2.0).

Here is the code from my unit tests, of course it's destined to change but I was in the mode of getting it to work with IronPython:

from Twining import *
import exceptions
import random
import unittest
import os

class TestSequenceFunctions(unittest.TestCase):

def setUp(self):
self.cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"

def test_exportcsv(self):
""" Export a CSV style data """
outPath = "c:\\foo.csv"
database(self.cn).table("Photos").copyto.csv(outPath)
self.assert_(os.path.exists(outPath), "File was not created")

def test_exportxmlattribstyle(self):
""" Export xml data attribute style """
outPath = "c:\\xml_as_attrib.xml"
database(self.cn).table("Photos").copyto.xmlFieldsAsAttributes(outPath)
self.assert_(os.path.exists(outPath), "File was not created")

def test_exportxmlelementstyle(self):
""" Export xml data attribute style """
outPath = "c:\\xml_as_elem.xml"
database(self.cn).table("Photos").copyto.xmlFieldsAsAttributes(outPath)
self.assert_(os.path.exists(outPath), "File was not created")

def tearDown(self):
if os.path.exists("c:\\foo.csv"): os.remove("c:\\foo.csv")
if os.path.exists("c:\\xml_as_attrib.xml"): os.remove("c:\\xml_as_attrib.xml")
if os.path.exists("c:\\xml_as_elem.xml"): os.remove("c:\\xml_as_elem.xml")

if __name__ == '__main__':
unittest.main()


You can download this and the latest version of things from the new Twining page


}

Monday, March 24, 2008

Twining Updates

{

What's the big picture and why are you doing it?

A comment from my last post got me thinking I need to get back to the big picture. Robert wrote:

Wow. That looks like a hard way to solve your problem.
If you absolutely have to mangle Excel data through Python and into a database... can I recommend Resolver? It won't cost you anything and it might just solve your problem.

I'll post about Resolver in a second but I want to go back to my original thoughts on developing Twining. I spend a lot of time doing quick and dirty types of data import and export.  Sometimes it's grabbing stuff off a production system so that I can develop off of it. Other times we get a simple file from someone and we need to pull it into the database.  Still other times we move data from one location to the next because we've done staging or cleanup. 

Traditionally the Microsoft tools for this are quite heavy.  DTS, which I used a lot, and SSIS which I haven't used much but have heard and read about are both GUI/IDE approaches towards the problem.  SSIS in particular isn't an out of the box solution for a developer like myself who doesn't always install every iota of SQL Server tool there is to install.  Usually I can express my problems in a simple sentence:

"Copy the data from this table to that table."
"Build an insert script for this table's data"
"Import this CSV file to this database"

I wondered why language couldn't be used as a tool for this type of work rather than launching a big GUI tool or writing throwaway C# console applications for the job.  That's the origin of the thought, cultivated when I heard Neal Ford give a talk at CodeMash on DSLs.  A single "sentence" lodged itself in my head and I thought, wow, that would be a nice solution:

database("foo").tables("bar").copyto.database("biff")

I have to admit to being more of a gardener than an architect when it comes to these things so I started a notepad file with some "forms" I thought would work and one friday afternoon, after being burned out on regular work I started what I call "sketching"; seeing how things might work. Since then I've been gnawing on it in a public way since learning Python and thinking in a Pythonic way are probably even more important than my little module which may end up just being fabulous meware.

I had posted that I had a little database import project of about 110,000 records distributed over two excel files.  The story I got was exactly the one I envisioned:

cn = "Data Source=.\\sqlexpress;Initial Catalog=MyDB;Integrated Security=SSPI;"
colmap = {"Field1":0, "Field2":5, "Field3":7} #destination:source - you may want to use the same source multiple times...
database(cn).table("DestinationTable").mapcolumns(colmap).importfrom.csv("C:\\temp\\widgetCo\\update1.csv")
database(cn).table("DestinationTable").mapcolumns(colmap).importfrom.csv("C:\\temp\\widgetCo\\update2.csv")

I will admit it wasn't perfect the first time - my lazy use of string concatenation rather than the .NET StringBuilder was a huge penalty (~10 minutes). I rewrote those pieces and the whole process took less than 30 seconds. Cheers for ADO.NET on that one - the updates were _massive.

So, back to Resolver; I read a glowing review from Larry O'Brien (his column is the first thing I read in SD Times when it crosses my desk) and the comment propelled me to watch a few screencasts and dig around but the goal of the product wasn't quite in line with the story I recount above.  Perhaps it's possible but it seems more like an interface in which one dwells on providing complex functionality in the spreadsheet rather than just quick and dirty import/export business.

More updates forthcoming but a short note: I spent a good part of the weekend trying to read through the code of ConfigObj 4 courtesy of Michael Foord (aka "Fuzzyman" aka "Voidspace").  I'll dwell on it further soon but one thing I realized I'd done that was a big mistake is to break the project into lots of files. I put them back together again and am in the process of figuring out how and where to host them. CodePlex/Google Code? Not sure if it's ready for that yet.  My own abode t3rse might work but I'd like to figure out how to generate documentation before I do that...

Enough said, here is what's latest, details and unit tests tomorrow.

}

Wednesday, March 19, 2008

Twining: Iron Python DSL for DB Update V

{

I'm still at it with Twining. A few projects and events have distracted me temporarily but I had a few moments last night to do a few things that I've intended to for some time.  First a summary of functionality with what's new in red.

cn = "Data Source=.\\sqlexpress;Initial Catalog=MyDB1;Integrated Security=SSPI;"
cn2 = "Data Source=.\\sqlexpress;Initial Catalog=MyDB2;Integrated Security=SSPI;"

# dest, source (you can use the same source column over and over)
# dest, source can be field names, example below uses ordinal position
colmap = {"DestinationField1":1, "DestinationField2":2, "DestinationField3":1}
database(cn).table("MyTable").mapcolumns(colmap).importfrom.csv("C:\\foo.csv")

#old stuff
mytrans = {"Field1":"upper","Field2": lambda x: x.upper(),"Field3": lambda x: x[::-1]}
database(cn).table("MyTable").transform(mytrans).copyto.database(cn2)
database(cn).table("MyTable").transform(mytrans).copyto.xmlFieldsAsAttributes("c:\\here.xml")
database(cn).table("MyTable").transform(mytrans).copyto.xmlFieldsAsElements("c:\\there.xml")
database(cn).table("MyTable").transform(mytrans).copyto.delimited("\t", "c:\\foaf.tsv")
database(cn).table("MyTable").copyto.delimited("\t", "c:\\foaf_no_trans.tsv")
database(cn).table("MyTable").transform(mytrans).copyto.csv("c:\\foo.csv")
database(cn).table("MyTable").copyto.xmlFieldsAsAttributes("c:\\here.xml")
database(cn).table("MyTable").copyto.xmlFieldsAsElements("c:\\there.xml")
database(cn).backupto("C:\\temp\\loungin.bak")

Footnotes: I'm working on changing my implementation of all things CSV/delimiter to using ADO.NET providers - the manually concatenated strings were just proof of concept. Back in 2004 in a thread that none other than Joel Spolsky responded to I learned not to try to reinvent a very hard CSV wheel.

The other bigger change I made was breaking the project apart. What was originally in multiple files as a "sketch" was becoming a nuisance besides poorly structured. I've deconstructed into the following files:

registrar.py - holds global, static data.
sqlhelper.py - runs dynamic sql statements
errorhandle.py - exception handling
database.py - reference to databases, constituent tables, and holds the copyto and importfrom objects
copyto.py - object for exporting as CSV, XML, and database to database
importfrom.py - object for importing data (currently on rough sketch for CSV import)

Most remains unchanged from the last upload but here is the code for the importfrom functionality:

import clr
clr.AddReference("System.Data")
from System import *
from System.Data import *
from System.Data.SqlClient import *
from System.Data.OleDb import *
from System.IO import *

from registrar import *
from sqlhelper import *


class importfrom:
"""Model data import process"""

def __init__(self):
self.columnmap = {}

def build_destination(self):
""" If destination does not exist, construct table"""
pass

def csv(self, path):
""" Import data from a csv file """
fileName = path.split("\\")[-1]
fileDir = path.replace(fileName, "")
connect = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties='text;HDR=No;FMT=Delimited'"
csvConnection = OleDbConnection(String.Format(connect, fileDir))
csvConnection.Open()
csvCommand = OleDbCommand("select * from " + fileName, csvConnection)
sourceReader = csvCommand.ExecuteReader(CommandBehavior.CloseConnection)
masterInsert = ""

while sourceReader.Read():
cols, vals = "", ""
for k in self.column_map.keys():
cols, vals = cols + "[" + k + "],", vals + "'" + str(sourceReader[self.column_map[k]]) + "',"
masterInsert += "insert into " + registrar.tableRegister + "(" + cols[:-1] + ")VALUES(" + vals[:-1] + ")\n"
sourceReader.Close()

try:
sqlhelper().run_sql(registrar.databaseRegister, masterInsert)
except Exception, inst:
errorhandle().print_error("Error occured during import", inst)

Download everything here as a zip file. I've hopefully carved a little space in the schedule amongst various projects to spend more time going back and polishing things. I've just been tasked with migrating about 120,000 records out of Excel so this should provide a good opportunity to add functionality and test it's breakability for the type of thing I often have to do.


}

Wednesday, March 12, 2008

Brash Boys

{

When I read the article on 37 Signals in Wired, I admit I alternated between shaking my head, laughing, and sighing.

When Hanson commented upon the term arrogance, I laughed:

"Arrogant is usually something you hurl at somebody as an insult," Hansson said. "But when I actually looked it up — having an aggravated sense of one's own importance or abilities' — I thought, sure."

When the article describes him as a "philosopher king" I shook my head.  And his response to naysayers, "F--- you" made me sigh; not a convincing piece of rhetoric at all. 

Despite all this, despite Atwood's recent swipe at what he calls "douchebaggery," despite the disagreement of Don Norman, there is something there. 

It's not that I'm one of the fanboys of the Ruby or Apple world, or that I agree with Hansson's attitude towards windows developers - it's that I think it's a trait of people I call successful to be able to concentrate and get things done without regard for outside opinions.  It's less of a "stay the course" mentality than an understanding of self and drawing clear lines with work rather than spending a lot of time second guessing and self doubting.

One area in which I can look at hard data and side with 37signals is in the realm of additional or complex features; I really do question whether they are worth it.  I just recently quantified this with an application we wrote for a customer that they are using successfully.  I compared the baseline functionality of a particular feature to some exceptional scenarios I had to write code for - it was easy because one can tell the usage by counting records in the database.  Anyhow, for the baseline functionality I counted some 6,100 instances of its use.  For the exceptional scenarios, I counted 9 entries.  Here's the clincher: it took about 3 or 4 times as long to implement support for the exceptional scenarios (and it added a lot of complexity to the code for the baseline feature).  So the ability to handle about one tenth of a percent of scenarios, cost and complexity was inflated significantly.

Here's where I disagree with them: it's not my decision, it's the customer's. My goal should be to communicate that trade off as well as I can.

}

Sunday, March 09, 2008

VB.NET Design

{

Paul Vick posts on some design considerations for future versions of VB.NET and hits upon something I was complaining about just this week - the fact that one has to speckle their code with line continuation characters in places where they should be implicit.  This seems to happen for me in two spots, assigning parameters and string concatenation. 

It's reassuring that not only does the VB.NET team listen, but they are looking for solutions as well.

}

Wednesday, March 05, 2008

Mix 08 Keynote

{

Just watched it while working on some stuff tonight.  Good things seem to be happening although much of it is in CTP or Beta form right now.  I'll need to resurrect my VPC images to install and play when I have the time. What's baking that's cool?

IE 8 - the integrated Firebug like debugging is what's got me excited for now.
Silverlight 2.0 - Dino Esposito joked about how Silverlight 1.0 was a declarative way to produce an animated gif (unless you just want a video player).  Now that a real control library is being developed, people like me will be able to seriously think of ways to use it.  I don't stream video but I do write a lot of code :)
Expression 2.0 / WPF - Ditto on Silverlight 2.0, WPF needs controls - and not just the kind you have to pay $$$ for.  In my best Steve Balmer voice I'd say "controls! controls! controls!" The blank canvas thing is cool but limited when you need to build something on a real world timeline with real world resources.

I didn't see ASP.NET MVC stuff in the keynote under demonstration but it was mentioned. Via Hanselman, Preview 2 is released today.

John Lam seems to be writing a multi-part article on DLR + IronRuby - a cool way to experiment with Silverlight although now I'm officially in love with Python and may not have room for a while. 

I'm excited to see what happens but I'm skeptical to put too much effort into this stuff until it's properly released.  Since I'm no longer a trainer and the apps I write have to be installable on client machines, I have less flexibility and time to get full fledged into beta stuff. It will also be interesting to see who wins the Silverlight / Flash war (and a war it will be). Microsoft is building its installed base pretty fast and with a better developer toolkit I will be interested to see what Adobe does with Flash.

Lots more lurking to do - even though all the cool people are at MIX, so much is online and so many people are twittering (I now have a "Twit" folder in my reader), one can almost imagine themselves there.

}

A Gem in the Morning Coffee

{

Harry "DevHawk" Pierson comments (emphasis mine):

My opinion, since you asked Nick, is that EA (Enterprise Architecture) fails to deliver value because it tries to control the uncontrollable. Trying to gain efficiency thru establishing standards and eliminating overlap via reuse are pipe dreams, though literally millions of $$$ have been poured into those sink-holes. There are a few areas where centrally funded infrastructure projects can solve big problems that individual projects can't effectively tackle on their own. EA should focus their time there, they can actually make a difference. Otherwise, they should stay out of project's way.

I concur and there's follow up if you read on in his blog, but this was quite succinct.

A funny little aside: I was called for a marketing survey and agreed to it after I found the reward was a $50 coupon for Amazon. The woman caller asked my title and I said "Software Architect" (my given title, though I really fit the mould of a lead developer and plan to have it changed).  She paused on "architect" and asked: "But do you write code?"  I answered in the affirmative and had to do so to variants of the question (e.g. "But do you implement specifications in code? But do you really know C#?") about 4 or 5 times.  Moral of the story, don't use architect in your title, use developer. Unless you're Don Box.

}

Just Do It Yourself

{

I go back and forth on the very thing Larry calls "Lead-Developer Compression Ratios." Two things happen over the long haul that affect my thinking: first, even though it's time consuming communicating needs to a junior developer, it seems to pay off when it comes to fine tuning and bug fixing - it's not something you have to worry about since it's delegated. "Finished" in my world means it's been through a few cycles of testing, cycles that take a lot of time.  The second thing though is when something is delegated and I think is done fairly well only to find at the aforementioned testing/bug fix cycle is horribly broken and either has to be hacked into a technical debt ridden working order or needs to be done again - from scratch.  It seems to come down to the complexity of the tasks - if it's simple you can take the time, if it's complex just do it yourself.

}

Monday, March 03, 2008

Twining: Iron Python DSL for DB Update IV

{

long entry because it was written at an airport, just back from celebrating 3 years with my beautiful wife in Monterey, CA...

A few interesting things, but first just a summary of what can be done with what's new in red:

#connection strings
cn = "Data Source=.\\sqlexpress;Initial Catalog=DB_SOURCE;Integrated Security=SSPI;"
cn2 = "Data Source=.\\sqlexpress;Initial Catalog=DB_DESTINATION;Integrated Security=SSPI;"

#back up the database to a file
database(cn).backupto("C:\\temp\\loungin.bak")

#copy/export basic
database(cn).table("MY_TABLE").copyto.csv("c:\\spam.csv")
database(cn).table("MY_TABLE").copyto.xmlFieldsAsAttributes("c:\\spam.xml")
database(cn).table("MY_TABLE").copyto.xmlFieldsAsElements("c:\\spam.xml")

#pick your delimiter
database(cn).table("Photos").copyto.delimited("\t", "c:\\spam.tsv")

#copy to database
database(cn).table("MY_TABLE").copyto.database(cn2)

#all copy operations can leverage transform dictionary
mytrans = {"FIELD1":"upper","FIELD2": lambda x: x.upper(),"FIELD3": lambda x: x[::-1]}

database(cn).table("MY_TABLE").transform(mytrans).copyto.xmlFieldsAsAttributes("c:\\spam.xml")
database(cn).table("MY_TABLE").transform(mytrans).copyto.xmlFieldsAsElements("c:\\spam.xml")
database(cn).table("MY_TABLE").transform(mytrans).copyto.delimited("\t", "c:\\spam.tsv")
database(cn).table("MY_TABLE").transform(mytrans).copyto.database(cn2)

... and so on, see above

Some internal changes:
1. Error Handling, quite lazy for now simply stores a combination of a friendly message along with full text of the exception thrown:

class errorhandle:
    def print_error(self, text, error):
        """ Handle an exception by just printing a friendly message and then the error's guts"""
        print text
        print error

try:
    sqlhelper().run_sql(registrar.databaseRegister, sqlBackup)
except Exception, inst:
    errorhandle().print_error("Error occured during backup", inst)

2. Some Pythonic prettiness (via Catherine):

#if registrar.transforms.has_key(field): -- es muy fao!
if field in registrar.transforms:       

I do like the more idiomatic form myself.  This a perfect example of where, as a .NET developer I look around for methods and miss a beautiful idiom.

3. Right under my nose, back to idiom, were a few simple pythonic forms that took away a lot of the line noise and clutter I was producing during my initial rabid sketches of the idea. Doing multiple assignment out of a SqlDataReader is something I only thought of now, though it makes such perfect sense:

column_name, column_type, column_length, column_identity = \
    r["name"], r["typename"], str(r["max_length"]), r["is_identity"]

4. Returning tuples rather than lists.  Tuples are more effecient and as effective.  A question that's begun burgeoning now is whether excessive use of tuples is lazy when a specific type is more clear.  For example, when I read column data from a table, I need to know two things: the column name and whether it's an identity (autonumbering). Right now I'm doing the following lazy gesture:

colTypeInfo.append((column_name, column_identity)) #(column_name, column_identity) is the tuple added to the list.

But if I start to add other useful bits of info, like the datatype of the underlying column, I wonder if a class would be better since you could have something like:

colTypeInfo.append(ColumnData(column_name, column_identity, column_type)) #Where ColumnData is a type somewhere

Right now the tuple approach works fine and a class feels like overkill. For now, I'm just afraid of it bloating as needs present themselves...

5. Speaking of identity columns, I added better support for them in tables being copied.  I also now use brackets for column names since certain people like to use reserved words for columns (or don't have a choice).

6. I mentioned in a comment that I tried to reconcile feedback from both Catherine and Michael "Fuzzyman" Foord.  When I match columns for transformations I allow any TypeError to be suppressed, but also store it in a "warnings" list that I'll incorporate into the completed process.

def checkAndTransform(self, field, value):
    """ Perform transformation on fields based on either:
        a) keyword entries ("upper", "lower", etc... )
        b) lambda expressions
    """
    if field in registrar.transforms:       
        if callable(registrar.transforms[field]):
            try:
                value = registrar.transforms[field](value)
            except TypeError:
                registrar.warnings.append("Your transform failed on field %s" % field)
        elif registrar.transforms[field] == "upper": #more of this kind of stuff _maybe later.
            value = value.upper()
    return value   

As always you can have a look at the entire thing here.  It's been great getting guidance, any comment is welcome and will be studied.

What is on the list?
1. I've been using a cheap, unstudied approach to delimited files. I'll need to look at the "official" way of escaping entries.
2. Importing data
3. A few more variants of operating on databases, like "create."

}

Tuesday, February 26, 2008

Twining: Iron Python DSL for DB Update III

{

My thoughts as exhibited in a little Python script now have a name: Twining. 

Unplanned illness slowed me down over the weekend, but the beauty of lambda expressions make transformations a breeze. I'll let the code document what I added.

cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
mytrans = {"Field1":"upper","Field1": lambda x: x.upper(),"Field3": lambda x: x[::-1]}
database(cn).table("Photos").transform(mytrans).copyto.csv("c:\\foo.csv")

As you can see, "mytrans" is a dictionary containing a matching of column names to both a keyword string, "upper", and also lambda expressions that model functions to operate on the constituent data (the [::-1] idiom reverses a string).  I'd add a few keywords for simple things people often do to data but the beauty of lambda expressions is that it keeps my code lazy and empowers whoever is using the script to make their own judgement about what to do with data.


Here's what goes on behind the scenes - the copyto class has a checkAndTransform method that matches up a field value with it's "transform":


	def checkAndTransform(self, field, value):
if registrar.transforms.has_key(field):
if repr(type(registrar.transforms[field])) == "<type 'function'>":
value = registrar.transforms[field](value)
if registrar.transforms[field] == "upper":
value = value.upper()
return value


I'm not Pythonic enough yet to know if I need to call repr and do a string comparison when testing if a type is a function, but it works just fine.


So a few thoughts:


1. What are the usual changes we make to data when exporting or importing?
2. Is it stupid to have shortcut strings or should I just fall back on writing lambdas?
3. Is there a better way of testing for a function than
repr(type(registrar.transforms[field])) == "<type 'function'>"


The to-do list from my previous post is still intact, I'll be working through it in the near future.  Have a look at the complete script and if you have thoughts or opinions, please let me know. A big goal here is not only to make something useful, but to learn Python (and be Pythonic, the two don't necessarily coincide).


Here's a full list of what can be done at this point:


mytrans = {"Field1":"upper","Field2": lambda x: x.upper(),"Field3": lambda x: x[::-1]}

database(cn).table("MyTable").transform(mytrans).copyto.csv("c:\\foo.csv") #with transform
database(cn).table("MyTable").copyto.csv("c:\\foo.csv") #without transform
database(cn).table("MyTable").copyto.database(cn2)
database(cn).table("MyTable").copyto.xmlFieldsAsAttributes("c:\\here.xml")
database(cn).table("MyTable").copyto.xmlFieldsAsElements("c:\\there.xml")
database(cn).backupto("C:\\temp\\my_backup.bak")



}

Thursday, February 21, 2008

Iron Python DSL for DB Update II

{

Attepting releasing early & often - here are a few new features to the ideas I'm putting together for the Database specific DSL.  I'll summarize all that's possible with it now with the new stuff highlighted in red.

#some connection strings
cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
cn2 = "Data Source=.\\sqlexpress;Initial Catalog=Irvine;Integrated Security=SSPI;"

#some things to do
database(cn).table("MyTable").copyto.csv("c:\\foo.csv")
database(cn).table("MyTable").copyto.database(cn2)
database(cn).table("MyTable").copyto.xmlFieldsAsAttributes("c:\\spam\\eggs.xml")
database(cn).table("MyTable").copyto.xmlFieldsAsElements("c:\\spam\\eggs.xml")
database(cn).backupto("C:\\spam\\eggs.bak")

The overall idea is to build a language centric, developer oriented implementation of what is done in a tool like SSIS or its predecessor DTS - I'm using SQL Server for a start but there's no reason this can't include other database platforms. It's quite exciting that even without knowing much Python I can get that much done in the language.  I'll spend time refactoring and reorganizing the script I've started with, but please feel free to download and modify it. Please post any improvements or ideas for how I can improve the Python and added any other features to the dsl. (Point on using __getattr__ well taken).

My todo list?
1. Aforementioned "transforms" for data in migration
2. Exception handling :^)
3. Copying tables is a rough sketch but needs a lot of work
4. Export formats besides CSV and XML: arbitrary delimiters and fixed field length
5. __getattr__ :^)

}

Tuesday, February 19, 2008

Iron Python DSL for ETL

{

Hanselman wrote about "fluent" interfaces recently which confirmed something I'd been thinking for a while. What if rather than massive tools we just had a language to express ourselves when we wanted to move data around?  What if we could say something like:

database("connection string").table("table").copyto.csv("C:\\spam\\eggs.csv")
or
database("connetion string").table("table").copyto.database("connection string 2")

In the spirit of releasing early and often, here's an implementation of the above language for sql server with Python. I'll be extending the flexibility quite a bit, and my Python skills are still those of a learner, but here it is for comment.  Before I post the whole script, here are a few goals:
1. XML data:
database("connection string").table("table").copyto.xml("C:\\spam\\eggs.xml")
2. Transforms:
transformDictionary = {'column1':'upper', 'column2':'some regex'} #these could become a fluent interface of their own.
database("connection string').table("table").withTransform(transformDictionary).copyto.csv("C:\\spam\\eggs.csv")

So here's that rather unrefined code download the latest implementation

Below was just some scratchwork from a Friday afternoon when I started sketching:

# BORRE DATABASE OBJECT
# 02/08/2008 - David Seruyange - @FRIDAY BURNOUT
import clr
clr.AddReference("System.Data")
from System import *
from System.Data import *
from System.Data.SqlClient import *
from System.IO import *

class registrar:
tableRegister = ""
databaseRegister = ""

class database:
"Model of a database somewhere"
def __init__(self, connectString):
"Initialize the rectangle"
self.connect = connectString
registrar.databaseRegister = connectString
self.copyto = copyto()

def table(self, t):
registrar.tableRegister = t
return self

class copyto:
"Model of a copy of some object"
def csv(self, path):
""" Save the result to a CSV file """
cn = SqlConnection(registrar.databaseRegister)
cn.Open()
cmd = SqlCommand("select * from " + registrar.tableRegister, cn)
r = cmd.ExecuteReader()
fileOut = open(path, 'w')
while r.Read():
sOut = ""
for i in range(r.FieldCount):
sOut += str(r[i]) + ","
sOut += "\n"
fileOut.write(sOut)
cn.Close()
fileOut.close()

def database(self, connect):
(ddl, colInfo) = buildDDL(registrar.tableRegister)
cnTarget = SqlConnection(connect)
cnTarget.Open()
cmdBuildTargetTable = SqlCommand(ddl, cnTarget)
cmdBuildTargetTable.ExecuteNonQuery() #builds the target table
cnSource = SqlConnection(registrar.databaseRegister)
cnSource.Open()
cmdSourceIter = SqlCommand("select * from " + registrar.tableRegister, cnSource)
sourceReader = cmdSourceIter.ExecuteReader()
insertCode = ""
while(sourceReader.Read()):
insertTemplate = "INSERT INTO " + registrar.tableRegister + " VALUES("
for i in range(sourceReader.FieldCount):
insertTemplate += "'" + str(sourceReader[i]) + "',"
insertCode += insertTemplate[:-1] + ")\n"
cmdExecuteInsert = SqlCommand(insertCode, cnTarget)
cmdExecuteInsert.ExecuteNonQuery()
# "save: %s\n table: %s\n to path: %s" % (registrar.databaseRegister, registrar.tableRegister, connect)

def buildDDL(tName):
query = """
select
c.[name], c.max_length, c.[precision], c.scale,
ty.[name] as typename
from
sys.columns c
join sys.tables t on c.object_id = t.object_id
join sys.types ty on c.system_type_id = ty.system_type_id
where
t.[name] = '__table__'
"""
query = query.replace('__table__', tName)
cn = SqlConnection(registrar.databaseRegister)
cn.Open()
cmd = SqlCommand(query, cn)
r = cmd.ExecuteReader()

ddl = """
CREATE TABLE __xyz__(
__columns__)
"""

columnsDefinition = ""
colTypeInfo = []
while r.Read():
#todo: support for other column types such as numeric or decimal
columnTemplate = " COLUMN_NAME DATATYPE LENGTH NULL,"
columnTemplate = columnTemplate.replace('COLUMN_NAME', str(r[0]))
columnTemplate = columnTemplate.replace('DATATYPE', str(r[4]))
colTypeInfo.append(str(r[4]))
if str(r[4]) == "varchar":
columnTemplate = columnTemplate.replace('LENGTH', "(" + str(r[1]) + ")")
elif str(r[4]) == "char":
columnTemplate = columnTemplate.replace('LENGTH', "(" + str(r[1]) + ")")
else:
columnTemplate = columnTemplate.replace('LENGTH', "")
columnsDefinition += columnTemplate + "\n"

r.Close()
cn.Close()
ddl = ddl.replace('__xyz__', tName).replace('__columns__', columnsDefinition).replace(",\n)", "\n\t)")
return [ddl, colTypeInfo]


cn = "Data Source=.\\sqlexpress;Initial Catalog=MVCBaby;Integrated Security=SSPI;"
cn2 = "Data Source=.\\sqlexpress;Initial Catalog=Irvine;Integrated Security=SSPI;"
# jepp
#database(cn).table("Photos").copyto.csv("c:\\foo.csv")
#print "---"
database(cn).table("Photos").copyto.database(cn2)

}

Monday, February 18, 2008

Release Early & Often

{

Paul Buchheit (who we can thank for Gmail) comments on a post by Marc Andreessen:

For web based products at least, there's another very powerful technique: release early and iterate. The sooner you can start testing your ideas, the sooner you can start fixing them.

I wrote the first version of Gmail in one day. It was not very impressive. All I did was stuff my own email into the Google Groups (Usenet) indexing engine. I sent it out to a few people for feedback, and they said that it was somewhat useful, but it would be better if it searched over their email instead of mine. That was version two. After I released that people started wanting the ability to respond to email as well. That was version three. That process went on for a couple of years inside of Google before we released to the world.

Interesting and it goes along with something Guy Kawasaki was saying in a recent podcast I commented on hearing.  So the next question comes down to technique - when you're working on something by yourself, how do you schedule builds?  I'm thinking I may start to take deliberate "coding vacations" - days I take off* from work in order to push through releases on my various little software projects.  Hm... other thoughts?

*My employer offers "comp days" so it's not at the expense of my day job.

}

Saturday, February 16, 2008

Check File Existence, Javascript (Prototype)

{

If you're like me, you spend a lot of time using System.IO and can easily check for the existence of a file on the filesystem with a File.Exists(@"C:\spam\eggs.file")

It's a little surprising that I hadn't run into it before but a few weeks ago I needed that same functionality with javascript. I found a few things online and what works best is using AJAX to make a request and check the HTTP status for whether or not the request failed.  My approach was to use Prototype and something like this:

function checkForFile(imgName, displayHolder){
new Ajax.Request('Repo/' + imgName + '.jpg',
{
method:'get',
onSuccess: function(transport){
var response = transport.status || "no response text";
if(parseInt(response) == 200){
displayHolder.src = 'Repo/' + imgName + '.jpg';
displayHolder.style.display='block';
}
},
onFailure: function(){
displayHolder.src = 'Repo/NoImage.gif';
displayHolder.style.display='block';
}
});
}

Works like a charm.


}

Mantras

{

Some good things to repeat to yourself from Dormando. I've a long way to go on so many of these and was geeky enough to give myself a scorecard.

I'll spare you that (after just deleting a lot of babble that would only be interesting to me) but there are two areas that I can do with some serious improvement:

1. Relentless automation
It's shameful how often I'm doing things I could easily automate (repeated FTP sessions?)
2. Monitoring
I wonder how out of line it would be to programmatically enable "breadcrumbs" for user's activity?  Finding out what they really used and what they insisted upon for nothing.

}

Saturday, February 09, 2008

Spitting Code Like Lam

{

I started changing my Visual Studio settings after seeing Hanselman's post and thought I'd be able to spit fire like John Lam once I'd downloaded his settings (it's gotta be the shoes!). The contrast was a bit much (though working in low light helped) and after a night's worth of being like JLam I wanted to go back to (dare I admit?) VS default settings. That didn't work out too well (even using the "default settings" option in VS, go figure) so I opted for Brad Wilson's settings which I've grown to like.

When the IDE "hot or not" thing first cropped up I changed settings but what makes me boring, preferring the default, is that I'm on a lot of machines at different times; my laptop, desktop, remote on a couple of servers, and when I help others.

}

Monday, February 04, 2008

Grabbing SQL 2005 Blobs with IronPython

{

On Saturday I needed to grab some blob data and create files out of it. The following was some Python I used, inspired very much by this code.

The biggest point of interest is the strongly typed array, which in IronPython is simple once you know how: using Array.CreateInstance.

 

 

#GET IMAGES FROM IMAGE BINARY FIELD
# 02/02/2008 - David - @THE COMMAND CENTRE

import clr
clr.AddReference("System.Data")
from System import *
from System.Data import *
from System.Data.SqlClient import *
from System.IO import *

CONNECT = "Data Source=(local)\sqlexpress;Initial Catalog=DATABASE;Integrated Security=SSPI"

sql = "select ID_FIELD, BLOB_FIELD from product_set"
cn = SqlConnection(CONNECT)
cn.Open()
cmd = SqlCommand(sql, cn)
reader = cmd.ExecuteReader(CommandBehavior.CloseConnection)
buffSize = 100
buff = Array.CreateInstance(Byte, buffSize)
recCounter = 0
while reader.Read():
print "Creating " + str(reader[0])
fs = FileStream(str(reader[0]) + str(recCounter) + "_diagram.png", FileMode.OpenOrCreate, FileAccess.Write)
bw = BinaryWriter(fs)
if(reader[1] != DBNull.Value):
posit = 0
b = reader.GetBytes(1, posit, buff, 0, buffSize)
while(b == buffSize):
bw.Write(buff)
bw.Flush()
posit = posit + buffSize
b = reader.GetBytes(1, posit, buff, 0, buffSize)
bw.Write(buff, 0, b)
bw.Flush()
bw.Close()
fs.Close()
recCounter = recCounter + 1

reader.Close()


I'd like to clean it up a bit, but I've written about 5,000,000 things that I "intend" to clean up before posting and they're never posted.


}

What Comes First; Rocks

{

Oliver Steele has a new essay entitled Adding the Easy Piece; or, The Metaphor of the Rock.  Insightful as usual, and I love the visualizations.

Personally, I'm experiencing the effects of pushing a rather large rock around a room.

}

Wednesday, January 30, 2008

Scott McCloud's Four Tribes - Developer Edition

{

Last year I was fortunate to catch Scott McCloud here in Sioux Falls. It's doubtful that he'd ever find his way here under normal circumstances but he was doing a 50 state tour in support of his book, "Making Comics."  I believe a dual aim was to provide his two young daughters, who travelled and presented with him, an experience of a lifetime. 

I was particularly struck with a classification he had for comic (and indeed all types) artists.  He placed them into four quadrants:

Here is a quote of an explanation of how the classifications work:

Those who value draftsmanship the most, and who are the most invested in the idea of mastery, beauty, and craft find themselves on the opposite side of the fence from those whose strongest values are raw honesty, authenticity, and a kind of rebellion against the status quo. That dichotomy comes between what I call the classicists and the iconoclasts. The children of Hal Foster versus the children of R. Crumb.

That’s the truth and beauty diagonal that separates two of the four tribes. The other two corners are the formalists and the animists, who represent the distinction between form and content. The formalists like to experiment and take the medium apart, figure out how it works and put it back together, and try new things; while the animists really just like to tell a compelling story, and want the form to vanish, to become transparent, so that you don’t know you’re reading a comic at all.

I wonder how well these notions apply to the practice of programming.  I can easily see the classicist: mastery, beauty, and craft - the person who uses things like recursion excessively for the computer science elegance they bestow.  Opposite them, I see the iconoclast -  the hacks that make things work and could care less about the principles or structure of a thing and instead devote themselves to the beauty of experience.

I think the animists and formalists can be distinguished as well by the metric of language/tool preference.  The obsessives of language would seem to tend toward formalism and understanding every moving part.  An animist may be the type of person who excels at making a tool do legwork with no particular interest in exactly what that tool is baking under the covers. 

Of course we all will have elements of each but I like to think of myself patterned after Scott McCloud. The formalist who might write the same software over and over in different languages just to see how expression and power vary at the lowest levels.

Interesting food for thought.

}

Tuesday, January 29, 2008

Java Programmers on Windows Forms team?

{

Randomly: worked with the CheckedListBox tonight. It's interesting to run into "getter/setter" type methods in the .NET universe.  One would assume a collection that specifically supported setting constituent objects as "selected" or "checked" but instead there are methods like: GetItemCheckedState and SetItemCheckedState.

It would be interesting to have the Windows Forms equivalent to the annotated base class library reference. Perhaps tomorrow I'll try to peak at the source code.

}

Disturbing News

{

"I am going to tell you something that will disturb you. You might laugh, but it will be a cold uncertain laugh that will haunt you as you read on, because somewhere deep down you'll know it to be true. You might brush it off, get on with your day, yet sometime later, a week or a year, it will seep back in and unsettle you to the core. From that moment on you will be changed. You will think different, act different and will fundamentally be different. So take a moment to prepare yourself now, breath deeply, clear your mind and open up to the possibility that building software is hard."

Interesting thoughts from The Wayward Weblog

 

}

Thursday, January 24, 2008

Not An Engineer

{

Ravi Mohan makes the provocative, if not new claim (with follow up) that programmers are not engineers. I take the majority of his quibble to be the title some flaunt: "software engineer."

I've never felt able to use "engineer" for myself, but I don't perceive this badly. My official title is "software architect" though most of my day is not spent on architecture.

It's a bit postmodern of me but does anyone's title mean much?  Short of very specialized work, one must have multiple skillsets for the different roles they are cast in on a daily basis.

I have to do the following, fairly regularly:
1. Write Code
2. Write Specs
3. Customer Service
4. Testing
5. Write Proposals
6. Mentor
7. Marketing

I'm owning up to a lot of stuff, particulary #7, that I would never admit to for the sake of saying that most jobs these days require multiple intelligences.

Think Santiago Calatrava: engineer, architect, artist. And he deals with clients, too.

}

Temporary Switch Back: C# to VB.NET

{

I did the switch back for a project that a VBA enlightened client requested but I kept finding myself asking how to syntactically do what I usually do in C# (especially inheritance-wise) in VB.NET.  Terry Smith has a handy online book that has a comparison so that you can easily figure out what the VB.NET syntax is for what you know how to do in C#. I'm not sure when it was written; no word on Generics but that's an easy enough syntax to duplicate with (Of T) rather than angle brackets.  What throws me off in addition to the inheritance verbosity are a lot of the array related casting operations that seem intuitive now in C# like:

Dim dropRows As Row() = e.Data.GetData(GetType(Row()))

Of course with the latest developments in VB.NET, I may end up using the language enough to make all of this easy to recall from memory. 

As much as I want to join the Red Sox fans in cheering for VB, there are still many things that annoy me about the language.  Line continuation characters are at the top of the list, and case insensitivity closely following. Yes, I strongly dislike case insensitivity in a programming language.

}

Tuesday, January 22, 2008

Up In The (Ruby) Club

{

A while back I had a phone call from a woman with an eastern European accent wondering if I would be willing to take a survey.  I'm usually an instant no to queries like this but she knew my Achilles heel:

"Participants get $50 Amazon Gift Certificate"

So a few days ago my inbox chimed with an email with the gift certificate in tow. I didn't hesitate with my purchase of David Flanagan's Ruby book

As an aside, it seems that Flanagan got some flack from a Gilles Bowkett on not being a bona fide Ruby person. If Flanagan isn't cultured enough to do Ruby, I'm in trouble...

... but I've been in trouble most of my short puff anyway.

On a serious note: the "up in the club" attitude of some (okay, mostly Rails) Ruby people is what is driving a backlash against Rails and Ruby in general.

}

Friday, January 18, 2008

Nutrition

{

I just looked at Oliver Steele's programmer's food pyramid and did a self evaluation of sorts. I know that I err too much on the reading of "blogs" versus reading long prose about programming.  That's something to change this year.

As well I use libraries a lot without digging into anything once my problem solving is done.  I think there's a form of mental stamina in digging deeper into a problem once you've apparently solved it - my instinct is to simply move on and try to tackle what's next.  Now that the .NET Framework code is available, that and other libraries should occupy my reading time.

While I don't struggle with the inclination to write code, I've long since thought it's better to try to write different types of applications and build some breadth.  Especially after my umpteenth web based CRUD app, I'm now interested in blending Windows Forms and Web stuff in the small variety of utilities that I compulsively write.

}

Wednesday, January 16, 2008

What's It Worth?

{

In the wake of all the plastic surgery and body augmentation going on around, a developer like myself can feel a little out of place - out of century to be more exact. 

And yet, whether he's joking or not, I'm half tempted by Wesner's admission to "performance enhancers."

But the thing I know with certainty is that being productive will only beget the desire to be more productive. Instead of learning and implementing 4 technologies, it will be 8.  I'll pile more home projects and "meware" until I'm aching for another 5 or 6 hours a day.

In a world that has the audacity to produce ridiculously smart people who are subject to feeling that yet others are "smarter" I may as well hold.

But it's tempting... maybe until I know F# and then I'll hold...

}

Saturday, January 12, 2008

13.75 hours, CodeMash

{

I earned the reputation as "the guy from South Dakota" at CodeMash; many people were amazed that I took it upon myself to drive just under 14 hours both there and back for the conference.

What would they have done if I told them I also:
a) took time off from work
b) paid for it out of pocket
c) drove my own car

But I think it's not *that* strange. Staying at work would have meant the same old stuff I do every day all year and instead I got to see some of the best Einstein's presenting on Python, DSLs, Dojo, REST, and IIS7. I made some friends as well and on top of that got to kick it with Scott Hanselman.  All for the cost of a long drive with an audio book.

I had posted on sessions I wanted to see, here's a little rundown of memorable sessions I actually attended:

1. Neal Ford's keynote "Software Engineering & Polyglot Programming"

Neal's one of those sedate cognoscenti who don't appear to have an emotional angle on platforms, just a sense of design and engineering that they wish to communicate.  In his talk he urged developers towards the notion of language as a tool, and that dynamic languages take us places that our old friends in the static realm cannot.  He made a reference to Jack Reeve's essay which I've written about before, and said a few times the following phrase which I thought was apt:
"Testing is the engineering rigor of software development."

If I could have asked Neal, I'd have wanted some comment on something I posted some time back - Oliver Steele's blog post on The IDE Divide in which he divides developers into the Language Mavens and Tool Mavens.  I think Neal's perspective was very much that of a language versus tool maven.

2. Catherine Devlin's "Crash, Smash, Kaboom Course in Python"
Great talk on Python fundamentals while illustrating a universe simulator library (forget the name).  I've been trying to pick up Python for some time and much of what I got was from watching Catherine's use of the language in her demos.  The one bummer: I was using a Python distribution from my iBook (Mac OSX) that didn't include the Python modules she was demonstrating with. Even more of a bummer: in my hotel room a different laptop was running Ubuntu with just the modules needed to follow along with her in code.

3. Kevin Dangoor on Dojo
I may have made Kevin feel uncomfortable with how happy I was to see this talk (he did a great job) but I've felt quite alone over the last two-ish years working with javascript libraries.  I used jQuery heavily for my tool "proper" after which I used Prototype/Scriptaculous for a few other projects.  I then switched to YUI for nRegex so I'd been around the block with my frameworks with only Dojo missing.  Dojo seems to go beyond these other frameworks just on its ambition alone and I'm wondering if that can translate into a happy mix with the heavyweight stuff I have to do at work with ASP.NET. 

4. Dustin Campbell on F#
Dustin gave a deft introduction to the F# language which had to be the biggest thoughtbludgeon I got during the conference.  I've listed F# as my "someday language" and it will remain so since Ruby will occupy my time along with Python in learning mode over the next year or so. Nevertheless, it was good to see a demonstration of the language and some of the implications thereof.

5. Kevin Dangoor on Gears and Dojo (DojoX Offline)
The second session I sat in on with Kevin, this dealing with Dojo's offline capabilities in large part courtesy of Google Gears.  It was as good as the first Dojo session and left my mind popping with ideas for leveraging that functionality. 
Soon - I'd say around the release of Silverlight 2.0 - people like me are going to face a fork in the road of using browser plugins and thick (they use the word "rich") vendor technologies, or using javascript and existing web technologies like CSS on a new level.  All the Dojo stuff represents, to me, what Microsoft and Adobe (no need to even mention JavaFX, oops I just did) are not tuning to as they ramp up Silverlight and Flex.  Not that it's in their interests: the stuff of web standards isn't really the profit center that proprietary technologies are.

6. Neal Ford on DSLs with Dynamic Languages
Neal was impressive this second time with some practical talk and examples of DSLs.  He started first with Groovy, which left me disappointed since I don't anticipate using it, but to show the "best" world he switched to Ruby.  I am really looking forward to IronRuby's release, but at present I think much of what he discussed can be implemented in Python. 

7. Dave Donaldson on RESTful Web Services
I've been familiar with the REST buzzword for some time but as a term thrown out in competition with standard web services (the WS-* stuff).  Dave did a good job of unpacking the essence of REST and showing demonstrations of usage.  It's interesting to see just how much backlash there is against WS-* and the manufactured complexity therein.  The only thing I disagreed with was Dave belittling Dave expressed reservations about the Astoria project teetering into the over-engineered realm of WS-* but I am hopeful it will retain its original intent - I really, really like the idea of Astoria (born of the same backlash to WS-*? I'm just talking here...).

8. Scott Hanselman on Mashups with IIS7
He has the gift and he knows it: first with a humorous prologue and then with a good presentation on IIS7 and using its underpinnings with a mix of PHP. Inspired by Bill Staples article on IIS7 and PHP, but with the detail and demonstrations with which Scott's earned his following.

I went to a few other sessions too, but these were the ones which, as the Brits say, "tickled my fancy."  I was disappointed to miss out on Jay Wren's intro to Castle as well as Bruce Eckel's thoughts on Python. I guess there's next year...

... and I do anticipate returning to CodeMash although I'm quite sure I'll fly rather than driving. :^)

}

Wednesday, January 02, 2008

Enemies and Organization

{

I usually try not to be an echo chamber, especially when posts show up on Coding Horror or Steve Yegge's blog. But what's interesting is a rebuttle to a notion that Yegge started, namely that size is an enemy in code base.

Franz Bouma has an interesting post to follow up that is worth a counter balance of views.

Conventions:

I commented once about LoC in one of our projects, not so much with an interest in maintainability, but as a comparison with how much code was being "generated" on my behalf by the IDE.  The project has grown since that post which had the codebase at around 30,000 lines of code.  This is nowhere near the 500,000 line mark but one design decision made early on that has always helped in navigating around was the heavy use of conventions.  Because we followed a pattern for where we put things, it's easy to find which project a piece of code is in as well as which part of the code one needs to dive into to make a modification. 

Abstraction:

One thing I'm noticing about the open source projects I look at is how layers of abstraction can lead to the kind of indirection that's hard to follow.  I speak only of ideals but it seems like too much abstraction can lead to the kind of confusing codebases that people online are reacting to. That logic may be the Mort in me going against the grain; it seems like many think that indirection and abstraction may add flexibility, but that flexibility will be a trade off for how easily a person, especially not the original author, finds their way.

}

Sunday, December 30, 2007

xPathFinder

{

The short version:
I'm releasing xPathFinder, a tool I wrote for doing XPATH expressions against xml files on the file system. You can get information about it as a tool by going to the website. Here are some screenshots:

 

The long version:
Not too long ago I ran into a problem that I thought would be common but didn't seem to have a canned solution. We use a webservice to store some data in a database but we also store the xml files transmitted on the file system as a defensive maneuver in case the database is unavailable or the xml is invalid. After a while I was dealing with a lot of xml files - far too many to search one at a time and needing more structured lookup than a desktop search tool offered. This is the raison d'etre of this tool. Of course it could have been a quick script in -insert language of choice- but this was something I needed to do repeatedly and have more facility with than console output. I'm also planning to make it simple enough for our client to use. It was a goal of mine to get it up before the end of the year so I'm happily posting this a day early.

One more footnote to this tool - stylistically I've done quite a bit of blending between Windows Forms and Web by leveraging the WebBrowser control heavily for things I thought would be easier to quickly generate html for as well as the feedback mechanism which allows users to report bugs or request features. It's so tightly coupled that while it will work without an internet connection, it's a much more limited experience. I really like this model and will probably start doing more blending like this for new work.

You can get more detail from the website or just ask directly by submitting feedback.

Future plans?

1. XPATH functionality works for most xml documents, but namespace heavy documents with prefixes seem to throw a curve ball.
2. What I'm posting was actually a fast prototype I intended to use to make this tool in WPF from - I'm not sure that the WPF step is still necessary but I may release an updated version since this is a fairly straight forward utility.
3. I'd like to make it open source. Not sure of what steps to follow, but CodePlex is probably the best starting point. I always promise myself I'll clean up my code before doing something like that but maybe I should just bite the bullet since I think the idea is sound enough.

}

Tuesday, December 11, 2007

Installing ASP.NET 3.5 MVC Preview

{

Code Climber has this great list of what you need to download and the order in which you must install it. I just completed my installation and have begun to poke around.

}

Thursday, December 06, 2007

Someday Language: F#

{

There's a lot to learn these days (whenever isn't there a lot?) and one thing I will do for 2008 is make a list of technologies I'd like to dive into with some sense of priorities. 

Among the ASP.NET MVC, Powershell 2.0, and Iron Python (for starters) I've been interested in F#. First as a topic on Hanselminutes but I lurk on Harry Pierson's blog and he's been posting about it quite a bit.  Here are some links:

1. F# Overview on hubFS
2. From DevHawk:

Someday it is.

}

Mads Samples

{

Mads Kristensen has a listing of his code samples. One thing I like about Mads' samples is that they are all very straightforward without unnecessary layers of abstraction.

}

Wednesday, December 05, 2007

The FogBugz Salesman

{

Once or twice people have confused what I'm doing with sales. Some time back I met with a prospective client and kept having to explain that I actually do write code and would probably be a lead developer on any work they gave us. The reaction I had for even the mere perception that I was "sales" was a little surprising to myself although with some reflection I realized it was because I conflated most of what I disliked about salesmen into an image I couldn't deal with for myself:

  • Style over substance
  • Shallow understanding
  • Pandering for a quick buck

But once in my life I was a happy salesman. Some time back I pushed the company I worked for into adopting FogBugz and really building a large part of our lifecycle around it. I did so enthusiastically after reading Joel's early essays on bug tracking and seeing how well it applied to our situation.

I've seen other bug tracking software since but still think the FogBugz experience was best.  It seems simple, but because of how much thought has gone into the product, I gather a lot of work has gone into keeping it stripped from unnecessary features.

To be clear, I don't work as a bona fide "in house" developer. But as a consulting company, I spend a lot of time playing that role or working with people in it.  It's therefore conflicting when I read his understanding of what it means to work in house. Phil Haack had it right in the middle of 2005 when he wrote of the irony of this disdain towards the people that would be excited to use his software in the first place.

I wonder what kind of conditioning it is - maybe that I've liked musical artists who are fairly open about loathing their fanbase - but I still miss the simplicity and elegance of FogBugz. 

What's sad these days is that as much as I'd like to slip on the poorly fitting suit and do my sales routine, we get Sharepoint for free (okay, not free but let's just say Microsoft does an excellent job making it seem so) and it's the environment in which we track what I'd gotten my previous company to use FogBugz for so effectively.  I'm not a big Sharepoint fan (that's an understatement) so I'd even be tempted to go The Powers That Be and ask for FogBugz if I couldn't predict the answer right away.

Tomorrow I'll be unflustered but I've got a few comments on what it's like to be in house:

1. In a small enough company, it's great to have decision making power.  I've used technologies like Ajax and Perl where a larger environment would have involved committee meetings with some manager shooting me down for doing things differently. You don't have to be a renegade to do this, just document what you do well and communicate with anyone you're working with that may have to look at it on your behalf.

2. In consulting it helps a lot to condition your client.  First build confidence with a track record of success, after which the client is usually willing to listen to you when you'd like to do things "right" or improve a piece of software even if it's getting by functionally.  After a while, depending on the client, they will usually just trust your judgement and stay away from technical arguments unless there's some disproportionate cost.

3. Software is never finished.  This is true even for what is developed in house.  As a result the process of continual improvement still exists.  It takes a tremendous amount of discipline to try to develop things right when there's a small budget or a tight deadline but I always think of those moments I've spent late at night hacking the ugliest workarounds because of a bad initial effort. Most of the time finishing when it's "good enough" is about the taste and craftsmanship of the person who writes the software; whether they tried to think of the future or not.

4. From working in both a "software company" and a "consulting company" I understand that software development is always a process of coming to terms with friction - all the unexpected realities and circumstances of real life and real users. Different places will have different kinds of friction; where an in house developer may have to deal with the kind of discipline it takes to write unit tests on a deadline a programmer at a software company may have the dilemma of trying to be all things to all people.

}

Tuesday, December 04, 2007

Comment Spam Begone

{

I maintain a few personal websites, one of the closest to my heart is phoDak, a photoblog.  I wrote the site using ASP.NET about two years ago and intended it to be a low key, personal area to post pictures.  I had absolutely no validation on comments and my "what will be" attitude worked for a long time until about a week ago when I saw a comment spam link to porn right off the most recent picture.  No longer avoidable, I looked into a few ways to block unwanted comments and came up with a hybrid approach:

1. Use a photo/word as validation for a human user, just like Jeff Atwood's blog.
2. I heard about Akismet from Phil Haack who mentioned it in passing on a Dot Net Rocks episode. I found a free library and wrote about 10 lines of code:

AkismetManager akismetManager = 
new AkismetManager("my key", "my site");
string ip =
Request.ServerVariables["REMOTE_ADDR"].ToString();
if(akismetManager.IsValidKey()){
AkismetItem item =
new AkismetItem(ip, Request.UserAgent.ToString());
item.AuthorName =
CommentOwnerTextbox.Text;
item.AuthorUrl =
CommentURLTextbox.Text;
item.Content =
CommentTextbox.Text;
if(akismetManager.IsSpam(item)){
akismetManager.SubmitSpam(item);
return;
}
}

Q.E.D., really. Somewhat Mort-ish for me to lean solely on the library but it works beautifully: my spam comments since implementation remain at absolute zero.


}

CodeMash: I'll be there!

{

On a whim I've signed up to be at CodeMash 2008.  After learning about the conference when Hanselman posted he'd keynote, I mapped out that Sandusky, OH (Whiskey, Tango, Foxtrot!) is none too far (an audio book's worth of distance).

I'm really looking forward to the opportunity to walk among Elvis and Einstein, hopefully gleaning some direction along the way. The session list is impressive, I'm thinking it will be hard to choose a particular direction.

I've been studying Python of late, so I'm really looking forward to Bruce Eckel's Why I Love Python. Another interestingly titled Python presentation Crash, Smash, Kaboom Course in Python will be run by MIT alum Catherine Devlin.  Finally on the Python front, I'll try to squeeze into the Getting Started with Django session.

On the "indy" .NET programming, I'm interested in the Introducing Castle session from Jay R. Wren. Because Hanselman is going to be there I can't imagine no mention of ASP.NET MVC stuff, so hopefully it will somehow be mentioned at least comparatively.

Jesse Liberty is going to present on Silverlight, so that's something I won't miss either. There are other Microsoft technology presentations I should see but it would be hard to sit in on a Sharepoint session when I could be learning Python design patterns.

The other area of interest I'll have is the Dojo stuff, two sessions in particular: Overview of the Dojo JavaScript Toolkit and Working Offline with Dojo+Google Gears, both by Kevin Dangoor.

Well, it should be interesting from start to finish, I'll be working on my note taking skills so that I can share what I do learn.

 

CodeMash – I&apos;ll be there!

}

Monday, December 03, 2007

Tis the Season

{

Don't get me wrong, I love Christmas too, but I really like the Advent Calendars that wind up the year. 

24 ways
Drew McLellan and his crew have been putting together web development tricks to round out the year for the last few years.

Perl Advent Calendar
None too late and looking like it was designed by a Perl programmer.

Two advent calendars I'd love to see:
1. Microsoft .NET
2. [Iron] Python

}