A Pandas DataFrame has a nice to_sql(table_name, sqlalchemy_engine) method that saves itself to a database.
The only trouble is that coming up with the SQLAlchemy Engine object is a little bit of a pain, and if you're using the IPython %sql magic, your %sql session already has an SQLAlchemy engine anyway. So I created a bogus
PERSIST pseudo-SQL command that simply calls
to_sql with the open database connection:
%sql PERSIST mydataframe
The result is (more...)
PyOhio gave my lightning talk on ddlgenerator a warm reception, and Brandon Lorenz got me thinking, and PyOhio sprints filled my with py-drenaline, and now ddlgenerator can inspect your data and spit out SQLAlchemy model definitions for you:
$ cat merovingians.yaml
name: Clovis I
name: Childebert I
$ ddlgenerator --inserts sqlalchemy merovingians.yaml
from sqlalchemy import create_engine, Column, Integer, Table, Unicode
I've had it on github for a while, but I finally released ddlgenerator to PyPI.
I've been frustrated for years that there was no good open-source way to set up RDBMS tables from flat data files. Sure, you could import the data - after setting up the DDL by hand. ddlgenerator handles that; in fact, you can go from zero, setting up and populating a table in a single line. Nothing up my sleeve:
Sometimes people want to connect to MySQL from Oracle and copy table data between the databases. You can do that with Oracle Hetrogenous Services via ODBC. This post will show how to create an odbc connection to your MySQL database which is the first part of this.
For my example, (more...)
I find myself installing and running mysql of different versions in different places for different reasons all the time (well often enough to do it and not remember the little things that cost time when setting up) Its with that in mind, I'm making notes for myself and you (more...)
For my newest ipython-sql trick, I needed to compare some queries run across different databases. How hard would it be to get side-by-side results into tidy IPython Notebook output?
Not hard at all, it turns out, if you're willing to violate basic principles of human decency.
That's an itty-bitty image, (more...)
When to consider sharding a MySQL database
Why should you not just design your applications from the start to account for data sharding? Sharding a database, that is horizontally partitioning tables among multiple databases, creates an added level of complexity that is not desirable in an application. However, there are times when sharding is required to maintain acceptable performance. So when is the right time to shard your database?
When to consider sharding a MySQL database Why should you not just design your applications from the start to account for data sharding? Sharding a database, that is horizontally partitioning tables among multiple databases, creates an added level of complexity that is not desirable in an application. However, there are times when sharding is […]
The MySQL documentation section has always had this Topic Guides page containing links to the docs for the various MySQL Connectors -- the official database drivers for various languages and programming technologies. That is the most convenient way to get the information for each Connector in PDF form, rather than downloading the entire Ref Man PDF. For HTML, it was more of a shortcut, because
As every year the DOAG (German Oracle User Group) Conference will take place Nürnberg/Germany from the 20th November until 22nd November 2012.
The Call for Papers is already open and ends on 30th June 2012:
German Link: http://www.doag.org/de/events/konferenzen/doag-2012/fuer-referenten.html
English Link: http://www.doag.org/en/events/konferenzen/doag-2012/fuer-referenten.html
Once again, I (more...)
This is THE
MySQL performance book. Period!
Every chapter is very well crafted, with a precise balance between theory and practice, and full of invaluable nuggets, sometimes transcending the MySQL arena and applicable to any database!
Such cases are Chapter 2-Benchmarking MySQL and Chapter 3-Profiling Server Performance, very solid foundations for the reading ahead.
All over the text, authors propose tools, examples of use and proven diagnostic techniques, that will greatly improve your performance firefighter skills and enhance your knowledge of MySQL internals. Nevertheless, what I liked the most from this book is taking into (more...)
Interesting take on what managers are thinking: http://quickbase.intuit.com/blog/2012/01/09/10-things-your-boss-isnt-telling-you/
After 30 years of Oracle, I've found myself using MySQL recently. I came across a little thing that surprised me. I'm by no means the first to trip over this - I found this 2006 post from Peter Zaitsey
on the same topic.
MySQL lets you write a group by statement that references columns that aren't in the group by, and aren't aggregates. For example:
mysql> select table_name, column_name, count(*)
-> from information_schema.columns
-> where table_schema = 'information_schema'
-> group by table_name
-> limit 5;
| table_name | column_name | count(*) |
| CHARACTER_SETS | CHARACTER_SET_NAME | (more...)
There was an interesting index related performance problem on Stack Overflow recently. The problem was to check an input string against a table that holds about 2000 prefix patterns (e.g.,
LIKE 'xyz%'). A fast select is needed that returns one row if any pattern matches the input string, or no row otherwise.
I believe my solution is worth a few extra words to explain it in more detail. Even though it’s a perfect fit for Use The Index, Luke it’s a little early to put it as an exercise there. It is, however, a very good complement (more...)
The InnoDB Plugin manual is now available on the MySQL web site.
I've been pretty quiet lately, because I'm in a transitional period. After 10 years on documentation for Oracle Database and other enterprise server products, I'm switching to the InnoDB group that already works with MySQL. New development environments, new customers, it's an exciting time!A decade seems to be the right timeframe for me. It was 10 years at IBM before that. Check back in 2019, I'm
In Europe, many people love Open Source with its connotations of sharing and village co-operatives. On the other hand, many Europeans don’t like the big, successful American companies (witness the ritual McDonalds-bashing).
This emotional preference for Open Source to big American companies is why the EU competition authorities have decided (more...)