I've never had a tool I really liked that would extract a chunk of a large production database for testing purposes while respecting the database's foreign keys. This past week I finally got to write one: rdbms-subsetter.
rdbms-subsetter postgresql://user:passwd@host/source_db postgresql://user:passwd@host/excerpted_db 0.001
Getting it to respect referential integrity "upward" - guaranteeing every needed parent record would be included for each child row - took less than a day. Trying to get it to also guarantee (more...)
A Pandas DataFrame has a nice to_sql(table_name, sqlalchemy_engine) method that saves itself to a database.
The only trouble is that coming up with the SQLAlchemy Engine object is a little bit of a pain, and if you're using the IPython %sql magic, your %sql session already has an SQLAlchemy engine anyway. So I created a bogus
PERSIST pseudo-SQL command that simply calls
to_sql with the open database connection:
%sql PERSIST mydataframe
The result is (more...)
PyOhio gave my lightning talk on ddlgenerator a warm reception, and Brandon Lorenz got me thinking, and PyOhio sprints filled my with py-drenaline, and now ddlgenerator can inspect your data and spit out SQLAlchemy model definitions for you:
$ cat merovingians.yaml
name: Clovis I
name: Childebert I
$ ddlgenerator --inserts sqlalchemy merovingians.yaml
from sqlalchemy import create_engine, Column, Integer, Table, Unicode
Yesterday was my first day at 18F!
What is 18F? We're a small, little-known government organization that works outside the usual channels to accomplish special projects. It involves black outfits and a lot of martial arts.
Kidding! Sort of. 18F is a new agency within the GSA that does citizen-focused work for other parts of the U.S. Government, working small, quick projects to make information more accessible. We're using all the tricks: small teams, (more...)
I've had it on github for a while, but I finally released ddlgenerator to PyPI.
I've been frustrated for years that there was no good open-source way to set up RDBMS tables from flat data files. Sure, you could import the data - after setting up the DDL by hand. ddlgenerator handles that; in fact, you can go from zero, setting up and populating a table in a single line. Nothing up my sleeve:
I went down a refactoring rabbit hole on ddl-generator and ended up pulling out the portion that pulls in data from various file formats. Perhaps it will be useful to others.
>>> from data_dispenser.sources import Source
>>> for row in Source('animals.csv'):
OrderedDict([('name', 'Alfred'), ('species', 'wart hog'), ('kg', '22'), ('notes', 'loves turnips')])
OrderedDict([('name', 'Gertrude'), ('species', 'polar bear'), ('kg', '312.7'), ('notes', 'deep thinker')])
OrderedDict([('name', 'Emily'), ('species', 'salamander'), ('kg', '0.3'), ('notes', '')])
tl;dr:Do not use public Google+ Hangouts under any circumstances, because people suck.
Before the PyCon 2014 CFP came due, PyLadies hosted several G+ hangouts for talk proposal brainstorming. Potential speakers could talk over and flesh out their ideas with each other, producing better talk proposals. More importantly, it was a nice psychological stepping stone on the way to filling out that big, scary CFP form all alone. I thought they went great.
I wanted (more...)
is programming language, that supports object-oriented, imperative and functional programming. The key is its simplicity, easy language to learn and easy moving code from development to production more quickly. It's power tool to use with Big Data. So, I believe it's a good time to learn about Python programming language.
You can find many resources about it on the Internet. I started to learn about Python by watching "Introduction to Python
" By Jessica McKellar. (more...)
Thanks to a certain recent Open SSL bug there’s been a lot of attention paid to passwords in the media. I’ve been using KeePassX to manage my passwords for the last few years so it’s easy for me to find accounts that I should update. It’s also a good opportunity to use stronger passwords than ‘banana’.
My problem is that I have always resisted the generation function in KeePassX because the resulting strings are very (more...)
Please consider participating in TRUCEConf (March 18-19 in Cincinnati)!
The goal is to help the tech community heal, through learning from others outside our industry and having an open dialogue and on how we can be better humans to each other in the world of tech.
You may remember fierce controversy around TRUCEConf when virtually nothing was known about it but its name; without solid information, it was easy to read bad connotations into the (more...)
After reading this fine summary of the history of Python 3 by Nick Coghlan I was inspired to update as many of my half finished projects and miscellaneous scripts as possible. Then I looked up and I had lost several hours of my life. To save random internet strangers from the same pain as I experienced here is a catalogue of problems and how I solved them.
tl;dr – Python 3.3 on Ubuntu 12. (more...)
I'm looking for collaborators who want to build web programming experience on an interesting project...
During my job search, I was contacted by Kai Schraml, a seminary graduate who wants to scratch an itch. Seminarians have a serious need to discuss, debate, and seek consensus on the translations of (more...)
I have just had a VERY. Busy. Week. (In a good way!) I've promised the world many talk materials, so:
As for Postgres Open, I absolutely loved (more...)
I'm happy to say that I'll shortly be starting a new position as a PostgreSQL DBA and Python developer for Zoro Tools!
We software types seem to have hardware envy sometimes. We have "builds" and "engines" and "forges" and "factory functions". But as it turns out, the "Tools" in "Zoro (more...)
Are you signed up yet for Ohio LinuxFest on Sep. 13-15? I'll be there to present
IPython for non-Pythonistas
Break out of your (bash) shell! IPython and the IPython Notebook have swept over the Python programming community, but they're not just for Python programmers - they make for high-powered shell (more...)
The Midwest Python Workshop for women and their friends is back! We've got new workshops scheduled, ready to take new batches of students:
Indianapolis Python Workshop, Sep. 27-28, 2013; sponsored by Six Feet Up and hosted at Launch Fishers
Columbus Python Workshop, Oct. 18-19, 2013; sponsored by LeadingEdje (more...)
I'm available for hire! If you need a database expert with lots of programming skill, or a Python programmer with deep database experience, please check out:
Resume / portfolio
But: you must be telecommute-friendly, or in the Dayton area. I'm sorry, but I'm not available to relocate.
At Monday's after-PyOhio sprint, I changed ipython-sql from an IPython Plugin to an Extension; this makes it compatible with IPython 1.0. Fortunately, this was really easy; mostly I just deleted Plugin code I didn't understand anyway.
But I do feel like "Writing Extensions" docs are lacking a "Hello World" (more...)
Loved PyOhio once again! Thanks so much to everybody who came, participated, and made it happen! I get such a rush of joy from seeing the Ohio Union fill up with happy Pythonistas.
PyOhio has been a classic case of the Stone Soup story. When we started planning the first (more...)
Among the many great talks
coming to PyOhio at the end of this month:
The IPython Notebook Revolution Catherine Devlin If you think of IPython as simply an enhanced version of the live Python prompt, you are in need of re-education. No matter what you do with Python, applying the (more...)