Do Python programmers just not care about handling errors? I’m using Python for a new project and am completely frustrated with the state of exceptions in Python. Python does so many things well. A simple thing like docstrings is brilliant and makes Python code much more useful, especially to those of us still somewhat new to the language (yes, we exist). But exceptions? They’re a big void, a black hole sucking away my time and sanity as I paw through pages and pages of documentation trying to find which ones are thrown for a given module or function.
Let me give you an example. I’m using SQLAlchemy for a project. I’m impressed with SQLAlchemy, it seems well designed. And the documentation seems ok, too. But, no. Let’s look at the query method. Quick, can you tell me what exceptions it throws? Well, if you hunt around a lot more, you eventually find this list of exceptions. But that doesn’t really tell me which ones query() can throw, and it doesn’t say that it’s an exhaustive list. Now look at all the code examples for SQLAlchemy. Any exception handling? I certainly haven’t found any. I guess errors never happen when using SQLAlchemy.
Let’s contrast this with the MySQL C API. Look at the page for query. I know exactly what errors occur and why for that function. Now let’s look at an example of how to use the C API. I picked a random example from a quick Google search. See that? All that sexy error checking!
Look, I know, documentation sucks. For every language and every library. It sucks to write, it sucks to maintain. And I don’t want to pick on just SQLAlchemy. Like I said, I like that project. But as I look through the documentation and example code for various Python modules, I want to scream. Apparently, people documenting Python code are pathologically incapable of explaining which exceptions get thrown and when. It’s like they live with the unicorns and LOGO programmers in the magic land of NoError ruled by King ICantProgram.
“Ok, so, yes, the documentation sucks, random self important blogger, but that’s not Python’s fault.” Wrong! They could have done something like docstrings, create a convention where it’d be easy to pull out which exceptions get called, and then also be able to propagate those up through the call stacks, so I could just query something in the interpreter and get a list back for any random function. It’d be so great, that I can’t even think of a good metaphor for its greatness. That’s how great it would be!
But that’s not where we’re at right now.</RANT>
Hey at least Python has exceptions.. try managing errors in a Perl program! I think the lack of clear Exception documentation is a consequence of the loose typing and generally scripty nature of Python work. If you want strict exceptions, etc, you want a stricter language like Java.
Perl’s built in exception handling mechanism is eval { die }. You can easily use Try::Tiny to get basic exceptions. There are other libraries available for other exception features.
Managing errors in well written code is easy. Managing them in poor code is painful. You can write both types in all languages.
Didn’t know SQLAlchemy is shipped with Python!
You’re making two mistakes here.
One is that you aren’t taking into account that there are two classes of exceptions in programming – application exceptions and business exceptions. These kinds of exceptions naturally fall under different documentation techniques, as they are handled in application code in two entirely different ways.
Application exceptions are the kind of exceptions where, when they get thrown, it means your application failed to function correctly. When this happens in SQLAlchemy, the exception will be in all cases a subclass of SQLAlchemyError, and if the exception came as a result of a DBAPI operation, it will be further a subclass of DBAPIError. Another subset of application errors are those which refer to API mis-use. These are InvalidRequestError and ArgumentError. Virtually every method in SQLAlchemy reserves the right to throw InvalidRequestError and ArgumentError when you do the wrong thing, not unlike how in Python itself there are tons of paths where ValueError, TypeError, and AttributeError might occur.
Since these exceptions can be raised in many scenarios, they would be necessarily present on *virtually every method’s documentation*.
The thing about application errors is, *you almost never write a catch for these kinds of exceptions*. They represent your app being broken, and should propagate all the way out to the outermost level where app failure can be tolerated. In a web framework, this is usually at the request level. You shouldn’t have a “catch” for InvalidRequestError, for example. You instead
want to have very generic handling for these errors at the highest levels of your application. Such as for a web application, at the request level. This is more of a programming technique which goes beyond the current scope of the SQLAlchemy docs, however there is a SQLAlchemy book project in the works that hopes to address techniques like these more completely. There are, FTR, a bunch of examples of catching application errors generically for the purposes of transaction handling, you can see them at http://docs.sqlalchemy.org/en/rel_0_7/core/connections.html#using-transactions, http://docs.sqlalchemy.org/en/rel_0_7/core/connections.html#nesting-of-transaction-blocks, and http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html#managing-transactions.
That said, it certainly would be very nice for all those methods which are known to invoke DBAPI operations be documented as such. I’ll address this in a moment, when I discuss the second mistake being made here.
But first, business exceptions. Unlike application exceptions, business exceptions are the kind of exceptions that are thrown in the normal course of action as part of the API of a method. These exceptions, you most certainly want to catch in certain situations. SQLAlchemy certainly documents these as they are expected, which mainly include http://docs.sqlalchemy.org/en/rel_0_7/orm/query.html#sqlalchemy.orm.query.Query.one, where you can get NoResultFound or MultipleResultsFound.
Overall, SQLAlchemy’s current approach to documenting exceptions is that one would read the pages you mentioned:
http://docs.sqlalchemy.org/en/rel_0_7/core/exceptions.html
http://docs.sqlalchemy.org/en/rel_0_7/orm/exceptions.html
where while reading these, one keeps in mind
that most of these exceptions can be thrown from an extremely wide variety of codepaths. All the exceptions that correspond to a flush can, for example, be thrown from the flush() method, the commit() method, as well as any situation in which the Query object autoflushes, which includes not just iteration of a Query object but any kind of lazy loading on an attribute – meaning *any attribute access can throw almost any of these exceptions*. This is OK, because these are application exceptions – you don’t catch them individually, they mean your application failed !
So for the second mistake. Yes, those “exceptions” pages can and should have more verbiage, more paragraphs describing SQLAlchemy’s approach to exceptions, when they are thrown and some general techniques for dealing with them. To which I will say, *your tax dollars are not paying for SQLAlchemy*. The documentation works out to many hundreds of pages and 90% of them were *written by me, personally, for free, on my own time*. The other 10% were written by other contributors. Which is the important part – *SQLAlchemy is an open source project*. All the effort you’ve expended writing this blog post could much better have been applied to *helping me write better documentation for exceptions*. That is how open source software works. Complaining about things is probably the lowest and most useless form of contribution, whereas *actually helping* is the highest. So to this I say,
*please stop blogging about the docs and help with the docs directly*. Patches/pull requests are *extremely welcome*, and this is *very clearly stated* on our site here: http://www.sqlalchemy.org/develop.html.
Looking forward to your help !
ZZZeek, thank you for your in depth reply. I now have a much better understanding of exceptions and SQLAlchemy. Like I said, I really like SQLAlchemy. And I hope we can contribute to your project. I was trying, perhaps poorly, to express my frustration about Python and exceptions in general and not just with a specific project. I apologize if it came off as anything else.