Hi! I’m Liz from Athenageek. Tarah and I are writing guest posts in each other’s blogs on the scripting languages we love.
I must make a confession before I continue with this post – I adore Python. It’s made me a better coder. I think about problems differently than I used to, I’m better at breaking them down into tiny components that are easy to code. My coding in other languages improved drastically as a direct result of learning Python – and using it in a Pythonic manner.
I used to use Bash for scripting. That’s what Bash is for, right? I use a Bash shell, so it only made sense.
One day, I was processing some xml scripts. I had multiple xml files with the same root node, and needed to put them all together into one big xml file. Frustrated at some sticking point or other, I decided to see if I could do it with Python. Oh, my goodness, it was SO EASY.
Before long, I had a string of utility files to handle common tasks that I was always scripting.
Reusability
When scripting, we often have tasks that we do over and over, but with slight changes. Rather than having a base set of methods that you copy out of and then change for each new usage, you can create utility files of classes and methods to use. Add methods to classes to handle new tasks, or allow new variables to be passed in for different usages. With Python’s named variables, you can change a method without effecting or needing to change other code that calls it.
One of the base tenants of Python is DRY (don’t repeat yourself). If you find yourself writing the same code more than once, put it in a place where it can be reused, like a method in a class or a function in a utility file.
Building up a libarary of these types of files is a great way to expand your toolbox. The longer you do it, the easier tasks become, because you’ve already done the work. Lots of scripting projects are, “Just like the one you did for x, but with this minor change”.
Testing
Python the community is heavily in favor of testing. One reason this is helpful for scripting is the case I mentioned above, when you change a method to handle some new functionality that you need. How do you make sure this new fuctionality doesn’t break old code? You test it. Doctesting and unit testing are built in to Python, no hunting around for which framework is best, balancing framework versions with script versions, or figuring out how to import your framework on a different operating system.
Doctesting
I love doctesting. In one fell swoop, you are showing how to use code, documenting that code, and proving that it works as expected.
Doctesting is very easy to do. Let’s say we create a method to square a number.
def square(numberIn):
return numberIn * numberIn
No documentation there, let’s fix that.
def square(numberIn):
"""Returns the square of the number passed in"""
return numberIn * numberIn
The doctest actually goes inside the documentation you’ve just created.
def square(numberIn):
"""Returns the square of the number passed in
>>> square(2)
4
"""
return numberIn * numberIn
So, you use >>> to delineate code that is going to be tested. That’s the same >>> you see when you’re working at a Python prompt. In this case, we’re calling our method and passing in 2. On the next line, put the expected result, which is 4. You can put as many of these tests as you like in the documentation block, separate them with a blank line. You can even put error checking here.
def square(numberIn):
"""Returns the square of the number passed in
>>> square(2)
4
>>> square('three')
Traceback (most recent call last):
...
TypeError: can't multiply sequence by non-int of type 'str'
"""
return numberIn * numberIn
I got that traceback by using my function at a command prompt. I copied and pasted it into my doctest. I deleted the variable stuff between the two important lines and replaced it with ellipses.
In order to run the doctest, you have to add the following to the bottom
of your file:
if '__main__' == __name__:
import doctest
doctest.register_optionflag('ELLIPSIS')
doctest.testmod()
Note that I’ve registered the ellipsis option flag. This allows you to use … as a wildcard so you can skip over some of the expected output. It keeps your code cleaner and lets you handle situations where expected output might be different – say – generating a random number.
Ok, now you can run the file. Let’s say we called it utility.py.
$ python utility.py
$
No output. That’s the default. In order to see the test results, you need to use the verbose flag.
$ python utility.py -v
Trying:
square(2)
Expecting:
4
ok
Trying:
square('three')
Expecting:
Traceback (most recent call last):
...
TypeError: can't multiply sequence by non-int of type 'str'
ok
1 items had no tests:
__main__
1 items passed all tests:
2 tests in __main__.square
2 tests in 2 items.
2 passed and 0 failed.
Test passed
$
And there are our test results – everything passed.
Aside from the option to put doctests in your code, you can also create txt files for documentation and add doctests to them. This is what bigger projects tend to do, especially when they have multiple files of code to deal with.
Unit tests are another great way to test that is very helpful for scripting. Every time you get a new requirement, add in a unit test for any situation you can think of that would effect that new requirement. Test things going right, things going wrong, and any unexpected behavior or input you can think of. If you find an error that isn’t covered in your tests, create a unit test to cover it. A baseline of code like this ensures that any future changes won’t break existing code. If it does, it will make the error really easy to track down – you won’t have to try to remember all the past requirements.
Easy for the next person to maintain
Between keeping code in one place, not repeating yourself, and having robust testing, you’ve just made maintaining this code really, really easy for the next person that comes along. Even if they don’t know Python, a glance at your robust documentation and clear code will make it easy to make small changes. Running the tests will leave them confident that their changes won’t have a negative effect on other code.
We all want to be able to go on vacation and not get a call that someone broke your code. Clear code with lots of tests are the best way to ensure a relaxing break. Python makes this quick and easy, so you can move on to something hard.
There’s a refrigerator that runs Linux now. That seems like a bit of overkill to me, but there are a few devices around the house that I think could use a bit of OSS ingenuity.
We have hibernation features for most laptops; is there a way to hibernate, say, a kitchen? Energy vampires like toasters (and I’m only going off of popular rumor, not fact here) are supposed to suck watts even when powered down. Wouldn’t it be handy to run a script that actually interfaces with your house’s power systems? I bet Bill Gates’ house does that already, so it’s time for the NIX crew to get it together 
My dream device: A kitchen touch screen computer like this one, but which can also be interfaced with remotely, and which can control the oven, the yogurt maker, the coffee pot, and the dishwasher. With that, I’d be able to interface my OurGroceries list with the kitchen’s DB on food stores, cook something remotely that requires a long cooking period with changes in temperature over time (like a crown roast), preheat the oven as I’m getting home to throw a pizza in, and which can warn me if the freezer or refrigerator rise above a certain temperature (a sure sign that the cats have found a way to knock open a door).
How about you?
There is nothing like configuring your own server to start getting a real understanding of how the guts of LAMP works. I’ve configured a lot of servers, but never built an entire Apache server from scratch before. Rackspace offers a cloud server for as little as $10 a month.
One of the first things you’ll want to do is create a server instance. I chose Ubuntu Maverick Meerkat (somewhat reflexively) as a server image. Rackspace builds the image and sends you an email when it’s done. You’ll use those credentials to login…but what next?
Though you can use the JavaScript console at Rackspace, the best move is to install SSH and create a user with sudoing privileges. This is where I started to get into areas that I wasn’t familiar with. I hadn’t configured SSH before, and Ubuntu creates users with the relevant permissions on graphical installation.
I would start giving instructions, but the truth is that I learned the most by tracking down the instructions myself. So, I’ll instead give a list of the things you’ll need to do.
- Install and configure SSH. Alter the standard port from 22 to something else so you don’t get hit by bots.
- Add a user with a home directory.
- Add that user to sudoers.
- Log out from the Rackspace console.
- Use either a bash shell from a NIX box or download Putty for use from a Windows machine.
- Krusader or Dolphin will work fine for file transport from NIX; download WinSCP for SFTP (secure file transfer protocol).
- Login using your shell and your new user. Use nano as a text editor for your work; it’s simplest. Install LAMP, Open SSL, and any other packages you want…but LAMP will do it. Use Tasksel–it’s the simplest way to install the full server.
- Use one of your spare domain names or a subdomain (blank.yourdomain.com), and point it to the IP address of your shiny new server.
- Configure your Apache Virtual Hosts (use sites-available and sites-enabled) to catch the incoming requests. That means to point requests for blank.yourdomain.com to a directory on your server, typically /var/www, but you can choose any. I’ve got sites in my home directory under individual names.
- Transfer files for any site using your SFTP or SCP protocols.
- Try hitting the site.
It’s rather simple, but it will take a while to configure. It’s fun to learn, but will be complicated, and is best treated as a hobby until you’ve got the security down. I’m also happy to answer questions.
If you’re like me, you can’t throw important documents away, but there’s no easy way to scan old records, tax documents, and all the crap you’re supposed to save for 7 or 10 years…or however long taxes are supposed to stick around. Unfortunately, batch scanning with Xsane using an ADF (auto document feeder) is not even remotely intuitive.
Here’s how to do it (I have an HP Officejet 6500, a relatively common all-in-one printer using hplip for drivers):
(1) Open Xsane. Choose the correct printer/scanner; it’s likely to be the top one in the list.
(2) There’s a tiny ticker field in the top left corner of the main Xsane window with no label on it. Set that to 40 or so (however many documents your ADF can take at a time, plus 5 or 10). You do that because if you leave it set at 1, Xsane will assume that you only want to scan the first document in the ADF. This way it knows to look for 40, and will quit afterwards. The other alternative is to set it at 30 or so, and push the scan button again after it’s stopped (if there are documents still in the ADF).
(3) Set the dropdown on the top right to “Multipage”. Ignore “batch scan” and the advanced settings.
(4) Create a working directory, and be sure to chmod and chown that directory recursively.*
(5) Create the multipage project in that directory. Set the output to PDF (if that’s what you want. PDF is probably best for archiving documents).
(6) Put a load of documents into your ADF, and hit ‘scan’ in the main Xsane window.
(7) Once they’re done, either load more and keep hitting the scan button to do batches of documents, or hit the “save multipage file” button at the bottom of the multipage project window.
Voila!
*You may have issues using the multipage project mode in Xsane if you don’t have proper read/write permissions set on your directory.
Open a terminal.
Enter at the prompt: sudo chmod -R a+rwx /yourHome/yourWorkingDirectory
Enter at the prompt: sudo chown -R yourUserName /yourHome/yourWorkingDirectory
Ok, so I am feeling guilty over not reading enough GOOD books.
So, I devised a cunning plan whereby I am emailed a portion of a classic book each day. I started with the Iliad. I know that I’ll clear my emails every single day because a cluttered inbox irritates me…so this is the perfect motivational tool. I wrote each of these scripts (with occasional help from the lovely folks who hang out with me in Programming Talk at the Ubuntu Forums), and this is the way I did this.
(1) Download a book you want to read. I suggest starting with something like the Iliad or Ovid’s Metamorphoses. Poetry is a great way to start; it’s episodic and fun. Head to Gutenberg.org to find text versions of any book you want. Other suggestions: the Aeneid, any Shakespeare play, the Divine Comedy, Paradise Lost, etc.
(2) Drop it into a folder, renaming it ‘[whatever].txt’. In my example, I’m dividing the Metamorphoses.
(3) Navigate to that folder at the CL.
(4) Run this script to divide the book up into 300 line segments (or, if there is no paragraph break at 300 lines, the script will add more lines until a paragraph break is reached. That way, you don’t end up getting half a soliloquy). Obviously, if you want to read more or less each day, change the number of lines to however much you want:
#!/bin/bash
bookcounter=0
linecounter=0
sed -i 's/\r$//' metamorphoses.txt
while read line
do
((linecounter+=1))
if [[ $linecounter -gt 300 && -z "$line" || -z "$output_file" ]]; then
linecounter=0
((bookcounter+=1))
formatted_bookcounter=$(printf "%03d" $bookcounter)
output_file=metamorphoses${formatted_bookcounter}.txt
echo "...starting segment $output_file"
echo "Metamorphoses - segment $formatted_bookcounter" > $output_file
echo "===================" >> $output_file
echo "" >> $output_file
fi
echo "$line" >> $output_file
done < metamorphoses.txt
(5) In your folder, delete the big file. You should now have several (or possibly several dozen or hundreds) of much smaller files, numbered consecutively.
(6) Then, run this script (Adds a subject header to each of the files in your folder) :
#!/bin/bash
FILES="*"
for f in $FILES ; do
sed -i '1s/^/Subject:Your Daily Book Part, Tarah.\n/' "$f"
cat "$f"
done
(7) Then, add this script to wherever you store your scripts. It sorts the files in your pet directory by number, gets the first one (i.e. ‘metamorphoses024.txt’) mails it to you, then deletes that file. Then, the next time the script is run, it will get ‘metamorphoses025.txt’ as the first file in the directory. Call it ‘bookmailer.sh’ or whatever. NB: you must have msmtp installed to make this work. Ensure it’s reachable through your path. Any other command line emailer will also work; I know how to use msmtp, so that’s what I do. Plus, I use Kubuntu, so it’s available at the repos.
#!/bin/bash
cd /home/tarahmarie/Documents/MyEbooks/aaa/
FILE=$(ls | sort -n | head -1)
cat $FILE | msmtp -a gmail youremailaddress@gmail.com
\rm `ls | sort -n | head -1`
(8) Execute ‘bookmailer.sh’ at the CL to ensure it works. If, in a few seconds, you get an email containing the subject line ‘Your Daily Book Part, WhoeverYouAre’, then the command is working. If you’re having any problems, ensure that you have permissions to each of these scripts, and that they’re marked ‘executable’.
(9) Add ‘bookmailer.sh’ to your system’s list of scheduled commands. I use kcron, so I set ‘bookmailer.sh’ to be executed each day at 8AM.
(10) Feel superior to other humans for the fact that you’re reading a ‘good’ book every day. This is DEFINITELY how I’m going to get all the way through War and Peace. Eventually. Let us not even speak of Remembrance of Things Past. I will CONQUER you, Proust, ole buddy. » Read the rest of this entry «
