Skip to content. | Skip to navigation

Personal tools
Log in
Sections
You are here: Home

Open Source Posts

Andrew Dunstan: Buildfarm bug fix for MSVC builds

From Planet PostgreSQL. Published on Apr 18, 2015.

Unfortunately there was a small bug in yesterday's buildfarm client release. The bug only affects MSVC builds, which would fail silently on the HEAD (master) branch.

There is a bug fix release available at http://www.pgbuildfarm.org/downloads/releases/build-farm-4_15_1.tgz or you can just pick up the fixed version of run_build.pl (the only thing changed) at https://raw.githubusercontent.com/PGBuildFarm/client-code/b80efc68c35ef8a1ced37b57b3d19a98b8ae5dd2/run_build.pl

Sorry for the inconvenience.

Baji Shaik: Aha, you can count the rows for \copy command.

From Planet PostgreSQL. Published on Apr 18, 2015.

We all know that \copy command does not return anything when you load the data. The idea is to capture how many # of records got loaded into table through \copy command.
Here's a shell script that should work:
echo number of rows in input: $(wc -l data.in)
( echo "\copy test from stdin delimiter '|';" ; cat data.in ) | psql -v ON_ERROR_STOP=1
echo psql exit code $?

If the exit code printed is 0, everything went well, and the value printed by the first echo can be used to to indicate how many rows were inserted. If the printed exit code is non-zero, no rows were inserted, of course. If the exit code printed is 3 then the data being copied had some error.

From the docs: If the exit code printed is 1 or 2 then something went wrong in psql (like it ran out of memory) or the server connection was broken, respectively. Following facts play a role in the above script:

.) COPY (and hence \copy) expects the input records to be terminated by a newline. So counting the number of newlines in the input is a reliable way of counting the records inserted.
.) psql will exit with code 3 iff there's an error in script and ON_ERROR_STOP is set. 
Note: This seems to not apply to the `psql -c "sql command"` construct.

# Example clean input

$ pgsql -c "create table test(a text,b int);"
CREATE TABLE
$ cat data.in
column1|2
column1|2
column1|2
column1|2
column1|2
column1|2

$ echo number of rows in input: $(wc -l data.in); ( echo "\copy test from stdin delimiter '|';" ; cat data.in  ) | psql -v ON_ERROR_STOP=1 ; echo psql exit code $?
number of rows in input: 6 data.in
psql exit code 0

# Example malformed input
$ cat data.in
column1|2
column1|2
column1|2c
column1|2
column1|2
column1|2

$ echo number of rows in input: $(wc -l data.in); ( echo "\copy test from stdin delimiter '|';" ; cat data.in  ) | pgsql -v ON_ERROR_STOP=1 ; echo psql exit code $?
number of rows in input: 6 data.in
ERROR:  invalid input syntax for integer: "2c"
CONTEXT:  COPY test, line 3, column b: "2c"
psql exit code 3
 
I hope this helps someone.

Baji Shaik: Woohoo !! Packt Publishing has published a book on troubleshooting PostgreSQL database.

From Planet PostgreSQL. Published on Apr 18, 2015.

(Baji is trying to impress 'X')
==========
Baji: Packt Publishing has published a book on troubleshooting PostgreSQL database.
 _X_: Uh, so what(!?). It published other 4 PostgreSQL books this year !
Baji: yeah, I know !
 _X_: then why do you care about thisssss.
Baji: I should care about it as I was part of technical reviewing team.. :(
 _X_: Oh really !, thats fantastic.. Congratulations !
==========

Note: Finally, Baji impressed _X_ :-)

Ok, in reality, I am glad to announce that "My first book as a Technical Reviewer has been published by Packt Publishing" ;-)

https://www.packtpub.com/big-data-and-business-intelligence/troubleshooting-postgresql
http://my.safaribooksonline.com/book/databases/postgresql/9781783555314/troubleshooting-postgresql/pr02_html

Author of this book is Hans-Jürgen Schönig, he has couple of other PostgreSQL Books as well.

This book is to provide a series of valuable troubleshooting solutions to database administrators responsible for maintaining a PostgreSQL database. It is aimed at PostgreSQL administrators who have developed an application with PostgreSQL, and need solutions to common administration problems they encounter when managing a database instance. So give a try ;-)

I would like to thank my loving parents for everything they did for me. Personal time always belongs to family, and I did this in my personal time.

I want to thank the Packt Publishing for giving me this opportunity and thanks to Sanchita Mandal and Paushali Desai for choosing me and working with me for this project.

Last but not least, would like to thanks Dinesh Kumar who taught me PostgreSQL and inspiring me for this. :)

Amit Kapila: Write Scalability in PostgreSQL

From Planet PostgreSQL. Published on Apr 17, 2015.


I have ran some benchmark tests to see the Write performance/scalability in 
PostgreSQL 9.5 and thought it would be good to share the same with others,
so writing this blog post.

I have ran a pgbench tests (TPC-B (sort of) load) to compare the performance
difference between different modes and scale factor in HEAD (e5f455f5) on
IBM POWER-8 having 24 cores, 192 hardware threads, 492GB RAM
and here are the performance result





























Some of the default settings used in all the tests are:
min_wal_size=15GB
max_wal_size=20GB
checkpoint_timeout    =35min
maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
autovacuum=off

I have kept auto vacuum as off to reduce the fluctuation due to same and is
dropping and re-creating the database after each run.  I have kept high values
of min_wal_size and max_wal_size to reduce the effect of checkpoints, probably
somewhat lower values could have served the purpose of this workload, but I
haven't tried it.

The data is mainly taken for 2 kind of modes (synchronous_commit = on | off) and
at 2 different scale factors to cover the cases when all the data fits in shared buffers
(scale_factor = 300) and when all the data can't fit in shared buffers, but can fit in
RAM (scale_factor = 3000).

First lets talk about synchronous_commit = off case, here when all the data fits in
shared_buffers (scale_factor = 300), we can see the scalability upto 64 client count
with TPS being approximately 75 percent higher at 64 client-count as compare to 8
client count which doesn't look bad. When all the data doesn't fit in shared buffers,
but fit in RAM (scale_factor = 3000), we can see scalability upto 32 client-count with
TPS being 64 percent higher than at 8 client-count and then it falls there on.

One major difference in case of Writes when data doesn't fit in shared_buffers is
that backends performing transactions needs to write the dirty buffers themselves
when they are not able to find a clean buffer to read the page, this can hamper
the TPS.

Now let's talk about synchronous_commit = on case, here when all the data fits in
shared_buffers (scale_factor = 300), we can see the scalability upto 64 client count
with TPS being approximately 189 percent higher at 64 client-count as compare to
8 client count which sounds good. When all the data doesn't fit in shared buffers,
but fit in RAM (scale_factor = 3000), we can see a pretty flat graph with some
scalability upto 16 client-count with TPS being approximately 22 percent higher than
at 8 client-count and then it stays as it is.

Here one point to note is that when the data fits in shared_buffers (scale_factor = 300),
TPS at higher client-count (64) in synchronous_commit = on mode becomes equivalent to 
TPS in synchronous_commit = off which suggests that there is no major contention
due to WAL writing in such loads.

In synchronous_commit = on case, when the data doesn't fit in shared_buffers 
(scale_factor = 3000), the TPS is quite low and one reason is that backends might
be performing writes themselves, but not sure if the performance is so low just
due to that reason as I have tried with different values of Bgwriter related parameters
(bgwriter_delay, bgwriter_lru_maxpages, bgwriter_lru_multiplier), but there is no
much difference.

As per my knowledge, the locks that can lead to contention for this workload
are:
a. ProcArrayLock (used for taking snapshot and at transaction commit)
b. WALWriteLock (used for performing WALWrites)
c. CLOGControlLock (used to read and write transaction status)
d. WALInsertLocks (used for writing data to WAL buffer)

I think among these ProcArrayLock and WALWriteLock are the candidates
which can be the reason for contention, but I haven't done any deep analysis
to find out the same.

Now it could be that the bottleneck is due to multiple locks as was the case
for read operations which I have explained in my previous Read Scalability
blog or it could be due to one of these locks.  I think all this needs further
analysis and work. Thats all what I want to say for now.

Shaun M. Thomas: PG Phriday: Anonymous Blocks and Object Manipulation

From Planet PostgreSQL. Published on Apr 17, 2015.

PGDB has had anonymous blocks since the release of 9.0 in late 2010. But it must either be one of those features that got lost in the shuffle, or is otherwise considered too advanced, because I rarely see it used in the wild. If that’s the case, it’s a great shame considering the raw power it conveys. Without committing to a function, we can essentially execute any code in the database, with or without SQL input.

Why is that good? One potential element of overhead when communicating with a database is network transfer. If processing millions of rows, forcing PGDB to allocate and push those results over the network will be much slower than manipulating them locally within the database itself. However, the looming specter of ad-hoc scripts is always a threat as well.

It was the latter scenario that prompted this particular discussion. A few weeks ago, I addressed date-based constraints and how they’re easy to get wrong. Knowing this, there’s a good chance we have objects in our database that need revision in order to operate properly. In one particular instance, I needed to correct over 800 existing check constraints an automated system built over the last year.

I hope you can imagine that’s not something I would want to do by hand. So it was a great opportunity to invoke an anonymous block, because there’s very little chance I’d need to do this regularly enough to justify a fully-fledged function. In the end, I came up with something like this:

DO $$
DECLARE
  chk TEXT;
  col TEXT;
  edate DATE;
  sdate DATE;
  tab TEXT;
  ym TEXT;
BEGIN
  FOR tab, chk, col IN 
      SELECT i.inhrelid::REGCLASS::TEXT AS tab,
             co.conname AS cname,
             substring(co.consrc FROM '\w+') AS col
        FROM pg_inherits i
        JOIN pg_constraint co ON (co.conrelid = i.inhrelid)
       WHERE co.contype = 'c'
  LOOP
    ym := substring(tab FROM '......$');
    sdate := to_date(ym, 'YYYYMM01');
    edate := sdate + INTERVAL '1 mon';

    EXECUTE 'ALTER TABLE ' || tab || ' DROP CONSTRAINT ' ||
        quote_ident(chk);

    EXECUTE 'ALTER TABLE ' || tab || ' ADD CONSTRAINT ' ||
        quote_ident(chk) || ' CHECK (' ||
        quote_ident(col) || ' >= ' || quote_literal(sdate) ||
          ' AND ' ||
        quote_ident(col) || ' < ' || quote_literal(edate) || ')';
  END LOOP;
END;
$$ LANGUAGE plpgsql;

I didn’t just use a bunch of unnecessary variables for giggles. The original version of this block used a single RECORD and a subquery to collect all of the necessary substitutions in their calculated forms. However, I felt this discussion needed a simpler step-by-step logic. Now let’s discuss this rather large block of SQL, because it is a very interesting lesson in several different aspects of the PL/pgSQL procedural language.

If you didn’t already know, you can loop through SQL results in a FOR loop, and pull SELECT results into variables while doing so. This is fairly common knowledge, so I won’t dwell on it. We should however, examine the query itself:

SELECT i.inhrelid::REGCLASS::TEXT AS tab,
       co.conname AS cname,
       substring(co.consrc FROM '\w+') AS col
  FROM pg_inherits i
  JOIN pg_constraint co ON (co.conrelid = i.inhrelid)
 WHERE co.contype = 'c'

Here, we’re making use of system catalog tables that help PGDB manage table metadata. First comes pg_inherits for information on child partitions, since they’re extremely likely to inherit from some base table as suggested by the partition documentation. Next, we incorporate information from pg_constraint so we know the name of each check constraint (contype of ‘c’) to modify.

Regarding the SELECT block itself, there is admittedly some magic going on here. The REGCLASS type serves a dual purpose in PGDB. For one, it is compatible with the OID object identifier type used extensively in the catalog tables. And second, when cast to TEXT, it outputs the schema and object name it represents, based on the current namespace. This means that, no matter where we are in the database, we will get a full substitution of the object—wherever it lives.

In that same block, we also abuse the consrc field to obtain the name of the column used in the constraint. There’s probably a more official way to get this, but as it turns out, the \w wildcard will match any word character. By globbing with +, we essentially grab the first series of word characters in the check. It might not work with other check constraints, but date-based partition rules generally only have an upper and lower bound. For these, the first match gives us the column name, and we don’t care about the rest.

Within the loop itself, things are a bit more straight-forward. After a bit of variable juggling, we start by dropping the old check. It was malformed, so good riddance. Then we build the new constraint based on our desired start and end dates. Note the use of quote_literal here. By using this function, the date variables are converted to text and quoted as static values. The end result is a query like this:

ALTER TABLE some_child_201504
  ADD CONSTRAINT ck_insert_date_201504
CHECK (insert_date >= '2015-04-01' AND
       insert_date < '2015-05-01')

Because these static text values do not match the column type, PGDB will automatically cast them in the physical constraint it actually creates. This prevents the check type mismatches we wrestled with in the last article.

So ends this example of fixing some broken DDL with an ad-hoc anonymous block. In the past, it was fairly common for DBAs to write queries using concatenation to write the DDL commands in the SELECT section of the query. Then we would direct that output to a script, and execute it separately. In this particular case, we would need two scripts: one to drop the constraints, and another to re-create them. That approach is certainly an option for those still uncomfortable working with anonymous blocks or EXECUTE statements.

In the end, I always encourage exploring capabilities to their full extent. Dig into Server Programming documentation if you really want to learn more.

Andrew Dunstan: New PostgreSQL Buildfarm Client Release

From Planet PostgreSQL. Published on Apr 17, 2015.

I have just released version 4.15 of the PostgreSQL Buildfarm Client. Here's what's changed:
  • support the new location for pg_upgrade
  • support running tests of client programs
  • support building, installing and running testmodules
  • use a default ccache directory
  • improve logging when running pg_upgrade tests
  • handle odd location of Python3 regression tests
  • add timestamp to default log_line_prefix
  • make qw() errors in the config file fatal (helps detect errors)
  • minor bug fixes for web script settings.
  • allow for using linked git directories in non-master branches
The last item might need a little explanation.  Essentially this can reduce quite dramatically the amount of space required if you are building on more than one branch. Instead of keeping, say, 6 checked out repos for the current six tested branches, we keep one and link all the others to it. This works almost exactly the way git-new-workdir does (I stole the logic from there). This doesn't work in a couple of situations: if you are using Windows or if you are using git-reference. In these cases the new setting is simply ignored.

To enable this new setting in an existing installation, do the following after installing the new release:
  • in your config file, add this setting:
    git_use_workdirs => 1,
  • remove the pgsql directory in each branch directory other than HEAD
Another good thing to do in existing installations would be to add "%m" to the beginning of the log_line_prefix setting in extra_config stanza.

Enjoy!

Peter Eisentraut: Storing URIs in PostgreSQL

From Planet PostgreSQL. Published on Apr 16, 2015.

About two months ago, this happened:

And a few hours later:

It took a few more hours and days after this to refine some details, but I have now tagged the first release of this extension. Give it a try and let me know what you think. Bug reports and feature requests are welcome.

(I chose to name the data type uri instead of url, as originally suggested, because that is more correct and matches what the parsing library calls it. One could create a domain if one prefers the other name or if one wants to restrict the values to certain kinds of URIs or URLs.)

(If you are interested in storing email addresses, here is an idea.)

Automatically and accurately remove red eye from user uploaded photos

By Cloudinary Blog - Django from Django community aggregator: Community blog posts. Published on Apr 16, 2015.

Automatic red eye removal Red eye often happens due to the use of flash in low light conditions as the light hits the eye very quickly and into the retina. It then bounces off of the back of the eye and emits a red color due to the blood vessels there. Although more professional modern cameras and flashes generally prevent this from happening, red eye may still occur with simpler, smaller cameras (including smartphones). There are various software solutions for red eye removal available on mobile devices and desktops, some of which require manual processing to get good results.

Obviously, it would be much faster and more convenient if this process were fully automatic, especially when dealing with a bulk of images that is uploaded by your web or mobile application’s users.

Cloudinary allows developers to automate red eye removal for websites and web applications. This especially comes in handy for social networks where users want their uploaded pictures to look as good as possible when they are shared among their family and friends.

How-to perform red eye removal

Cloudinary's rich manipulation capabilities allow you to further enhance users’ uploaded photos with options such as face detection-based cropping, resizing and rotating, increasing color saturation and more. With this new capability incorporated into Cloudinary’s image lifecycle management, developers can automate red eye removal by setting the effect parameter within Cloudinary's dynamic manipulation URLs to redeye. This enables smart red eye removal algorithms to be automatically applied on-the-fly to uploaded images.  

In the example below, the image on the left shows a scaled down version of an original image with red eyes and the image on the right shows a scaled down version of the original image with Cloudinary’s red eye removal feature dynamically applied.

Ruby:
cl_image_tag("itaib_redeye_msjmif.jpg", :effect=>"redeye")
PHP:
cl_image_tag("itaib_redeye_msjmif.jpg", array("effect"=>"redeye"))
Python:
CloudinaryImage("itaib_redeye_msjmif.jpg").image(effect="redeye")
Node.js:
cloudinary.image("itaib_redeye_msjmif.jpg", {effect: "redeye"})
Java:
cloudinary.url().transformation(new Transformation().effect("redeye")).imageTag("itaib_redeye_msjmif.jpg")
jQuery:
$.cloudinary.image("itaib_redeye_msjmif.jpg", {effect: "redeye"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Effect("redeye")).BuildImageTag("itaib_redeye_msjmif.jpg")

Original image

Uploaded image with red eye removed

Here we used the same images as above (before red eye removal and after) to generate face detection based thumbnails. This, as well as red eye removal, can be done by embedding a dynamic manipulation URL and code (as shown below) from various development frameworks into your web page.

Ruby:
cl_image_tag("itaib_redeye_msjmif.jpg", :transformation=>[
  {:effect=>"redeye"},
  {:radius=>"max", :width=>200, :height=>200, :crop=>:thumb, :gravity=>:face}
  ])
PHP:
cl_image_tag("itaib_redeye_msjmif.jpg", array("transformation"=>array(
  array("effect"=>"redeye"),
  array("radius"=>"max", "width"=>200, "height"=>200, "crop"=>"thumb", "gravity"=>"face")
  )))
Python:
CloudinaryImage("itaib_redeye_msjmif.jpg").image(transformation=[
  {"effect": "redeye"},
  {"radius": "max", "width": 200, "height": 200, "crop": "thumb", "gravity": "face"}
  ])
Node.js:
cloudinary.image("itaib_redeye_msjmif.jpg", {transformation: [
  {effect: "redeye"},
  {radius: "max", width: 200, height: 200, crop: "thumb", gravity: "face"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .effect("redeye").chain()
  .radius("max").width(200).height(200).crop("thumb").gravity("face")).imageTag("itaib_redeye_msjmif.jpg")
jQuery:
$.cloudinary.image("itaib_redeye_msjmif.jpg", {transformation: [
  {effect: "redeye"},
  {radius: "max", width: 200, height: 200, crop: "thumb", gravity: "face"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Effect("redeye").Chain()
  .Radius("max").Width(200).Height(200).Crop("thumb").Gravity("face")).BuildImageTag("itaib_redeye_msjmif.jpg")

Original thumbnail

Face detection based thumbnail with red-eye removed

Leveraging eye detection for more accurate red eye removal

In order to get even higher quality results, you can use Cloudinary’s ReKognition face attribute detection add-on for eye detection. Together with this add-on and the red eye removal effect, Cloudinary can automatically detect where eyes are located in a photo and apply the red eye removal algorithm in a more precise way. In order to do this, set the effect parameter of Cloudinary’s dynamic manipulation URLs to rek_redeye. Cloudinary's SDKs allow you to easily generate manipulation and delivery URLs in various development frameworks. Below is a sample dynamic manipulation URL and code to generate an HTML image tag that can be adjusted for various popular frameworks such as Ruby on Rails, PHP, Node.js, and more.

Following the examples above that simply underwent dynamic red eye removal, below is an original uploaded image that was cropped and underwent accurate red eye removal using Cloudinary’s ReKognition face attribute detection add-on.

Ruby:
cl_image_tag("tali_redeye_rvem1u.jpg", :effect=>"rek_redeye")
PHP:
cl_image_tag("tali_redeye_rvem1u.jpg", array("effect"=>"rek_redeye"))
Python:
CloudinaryImage("tali_redeye_rvem1u.jpg").image(effect="rek_redeye")
Node.js:
cloudinary.image("tali_redeye_rvem1u.jpg", {effect: "rek_redeye"})
Java:
cloudinary.url().transformation(new Transformation().effect("rek_redeye")).imageTag("tali_redeye_rvem1u.jpg")
jQuery:
$.cloudinary.image("tali_redeye_rvem1u.jpg", {effect: "rek_redeye"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Effect("rek_redeye")).BuildImageTag("tali_redeye_rvem1u.jpg")

Original image

Uploaded image with red eye removed using ReKognition eye detection

Final Notes

Cloudinary’s advanced image manipulation capabilities improve photo quality without any added effort on your side, and are fully integrated into Cloudinary's image management lifecycle. Simply add the parameters outlined above to an image’s CDN delivered URL and apply further effects, if desired, to adjust sharpness, color balance and more. The red eye removal feature is available with all of Cloudinary’s plans, including the free tier. You can use the ReKognition add-on eye detection effect by subscribing to the add-on itself. If you don't have a Cloudinary account yet, sign up for a free account here.

Josh Berkus: Expressions VS advanced aggregates

From Planet PostgreSQL. Published on Apr 16, 2015.

So ... you're using some of 9.4's new advanced aggregates, including FILTER and WITHIN GROUP.  You want to take some statistical samples of your data, including median, mode, and a count of validated rows.  However, your incoming data is floats and you want to store the samples as INTs, since the source data is actually whole numbers.  Also, COUNT(*) returns BIGINT by default, and you want to round it to INT as well.  So you do this:

    SELECT
        device_id,
        count(*)::INT as present,
        count(*)::INT FILTER (WHERE valid) as valid_count,
        mode()::INT WITHIN GROUP (order by val) as mode,
        percentile_disc(0.5)::INT WITHIN GROUP (order by val)
          as median
    FROM dataflow_0913
    GROUP BY device_id
    ORDER BY device_id;


And you get this unhelpful error message:

    ERROR:  syntax error at or near "FILTER"
    LINE 4:         count(*)::INT FILTER (WHERE valid)
            as valid_count,


And your first thought is that you're not using 9.4, or you got the filter clause wrong.  But that's not the problem.  The problem is that "aggregate() FILTER (where clause)" is a syntactical unit, and cannot be broken up by other expressions.  Hence the syntax error.  The correct expression is this one, with parens around the whole expression and then a cast to INT:

    SELECT
        device_id,
        count(*)::INT as present,
        (count(*) FILTER (WHERE valid))::INT as valid_count,
        (mode() WITHIN GROUP (order by val))::INT as mode,
        (percentile_disc(0.5) WITHIN GROUP (order by val))::INT
           as median
    FROM dataflow_0913
    GROUP BY device_id
    ORDER BY device_id;


If you don't understand this, and you use calculated expressions, you can get a worse result: one which does not produce and error but is nevertheless wrong.  For example, imagine that we were, for some dumb reason, calculating our own average over validated rows.  We might do this:

    SELECT
        device_id,
        sum(val)/count(*) FILTER (WHERE valid) as avg
    FROM dataflow_0913
    GROUP BY device_id
    ORDER BY device_id;


... which would execute successfully, but would give us the total of all rows divided by the count of validated rows. That's because the FILTER clause applies only to the COUNT, and not to the SUM.  If we actually wanted to calculate our own average, we'd have to do this:

    SELECT
        device_id,
        sum(val) FILTER (WHERE valid)
            / count(*) FILTER (WHERE valid) as avg
    FROM dataflow_0913
    GROUP BY device_id
    ORDER BY device_id;


Hopefully that helps everyone who is using the new aggregates to use them correctly and not get mysterious errors.  In the meantime, we can see about making the error messages more helpful.

Andreas Scherbaum: German-speaking PostgreSQL Conference 2015

From Planet PostgreSQL. Published on Apr 15, 2015.

Author
Andreas 'ads' Scherbaum

PGConf.de 2015 is the sequel of the highly successful German-speaking PostgreSQL Conferences 2011 and 2013. Due to space limitations in the old location, we are moving to Hamburg. The conference takes place on Friday, November 27th. We also add a day with trainings on the 26th.

http://2015.pgconf.de/

Registration for the conference will be possible well in advance. Tickets must be purchased online. For sponsors, we have put together a package that includes among other things, a number of discounted ticket. More in the Call for Sponsors in a separate announcement.

Andrew Dunstan: Hot Fix for buildfarm client, currently broken by pg_upgrade move

From Planet PostgreSQL. Published on Apr 15, 2015.

Yesterday the pg_upgrade program was moved from contrib to bin in the source tree. Unfortunately this broke most of those buildfarm members which check pg_upgrade. There is a hot fix for the TestUpgrade buildfarm module that can be downloaded from github. I will work on cutting a new buildfarm release in the next few days, but this file can just be dropped in place on existing installations.

Release 0.9.0

By Lightning Fast Shop from Django community aggregator: Community blog posts. Published on Apr 14, 2015.

We just released LFS 0.9.0. This is a bugfix release.

Changes

  • Replaced deprecated mimetype with content_type (Pigletto)
  • Added lfs_regenerate_thumbs command (Sasha Vincic)
  • Added links from property group property tab to properties
  • See README.txt for more changes.

Information

You can find more information and help on following locations:

Michael Paquier: Postgres 9.5 feature highlight: log_autovacuum_min_duration at relation level

From Planet PostgreSQL. Published on Apr 14, 2015.

log_autovacuum_min_duration is a system-wide parameter controlling a threshold from which autovacuum activity is logged in the system logs. Every person who has already worked on looking at a system where a given set of table is bloated has for sure already been annoyed by the fact that even a high value of log_autovacuum_min_duration offers no guarantee in reducing log spams of not-much-bloated tables whose autovacuum runtime takes more than the threshold value, making its activity being logged (and this is after working on such a lambda system that the author of this feature wrote a patch for it). Postgres 9.5 is coming with a new feature allowing to control this logging threshold at relation level, feature introduced by this commit:

commit: 4ff695b17d32a9c330952192dbc789d31a5e2f5e
author: Alvaro Herrera <alvherre@alvh.no-ip.org>
date: Fri, 3 Apr 2015 11:55:50 -0300
Add log_min_autovacuum_duration per-table option

This is useful to control autovacuum log volume, for situations where
monitoring only a set of tables is necessary.

Author: Michael Paquier
Reviewed by: A team led by Naoya Anzai (also including Akira Kurosawa,
Taiki Kondo, Huong Dangminh), Fujii Masao.

This parameter can be set via CREATE TABLE or ALTER TABLE, with default value being the one defined by the equivalent parameter at server-level, like that for example:

=# CREATE TABLE vac_table (a int) WITH (log_autovacuum_min_duration = 100);
CREATE TABLE
=# ALTER TABLE vac_table SET (log_autovacuum_min_duration = 200);
ALTER TABLE

Note that This parameter has no unit and cannot use any units like the other relation-level options, and it has a default unit of milliseconds, so after CREATE TABLE the autovacuum activity of relation vac_table is logged if its run has taken more than 100ms, and 200ms after ALTER TABLE.

Thinking wider, there are two basically cases where this parameter is useful, an inclusive and an exclusive case:

  • when system-wide log_autovacuum_min_duration is -1, meaning that all the autovacuum activity is ignored for all the relations, set this parameter to some value for a set of tables, and the autovacuum activity of this set of tables will be logged. This is useful to monitor how autovacuum is working on an inclusive set of tables, be it a single entry or more.
  • when willing to exclude the autovacuum runs of a set of tables with a value of log_autovacuum_min_duration positive, simply set the value for each relation of this set at a very high value, like a value a single autovacuum is sure to not take, and then the autovacuum activity of this set of tables will be removed from the system logs.

In short words, this parameter is going to make life easier for any person doing debugging of an application bloating tables, and just that is cool.

Travis and tox revisited

By dominicrodger.com from Django community aggregator: Community blog posts. Published on Apr 14, 2015.

Two years on, and I still love the combination of tox and Travis. I still write changes to my tox.ini and .travis.yml files separately, despite having written a tool for this. This tool was used a bit like this:

python tox2travis.py > .travis.yml

It occurred to me yesterday that there was a better way of writing this now - since tox now has a command for listing out what environments are set up (something which I think didn't exist when I wrote the original Python script).

Rewritten in bash, it now looks like this:

#!/bin/bash
set -o nounset
set -o errexit

command -v tox >/dev/null 2>&1 || { echo >&2 "tox2travis requires tox but it's not installed. Aborting."; exit 1; }

if [ ! -f tox.ini ]; then
    echo "tox.ini not found. Aborting."
    exit 1
fi

echo "language: python"
echo "python: 2.7"
echo "env:"

for env in $(tox -l); do
    echo "  - TOX_ENV=${env}"
done

echo "install:"
echo "  - pip install tox"

echo "script:"
echo "  - tox -e \$TOX_ENV"

This is functionally pretty identical (though a little better at error checking) to my previous effort, but is a good deal neater, and doesn't depend on tox internals.

I've also set it up so that it's on my path, which is the main (stupid) reason why I never really got used to using the old version.

Rajeev Rastogi: Indian PGDay, Bangalore (11th April 2015)

From Planet PostgreSQL. Published on Apr 13, 2015.

I recently just got back from Indian PGDay conference 2015. It was an interesting, motivating and lot of knowledge sharing in terms of both attending and speaking at the conference.

I spoke about the various kind of "Optimizer Hint" provided by many database engines and also a new idea of "Hint", which can be provided to Optimizer. Some of the speakers shared their work on PostgreSQL as User.
Also it was interesting to know that many companies are evaluating migration or are in process of migrating from other DBs to PostgreSQL. This is really encouraging for all PostgreSQL experts.



Some of the details from presentation are as below (For complete presentation please visit Full Optimizer Hint)

Statistics Hint:

Statistics Hint is used to provide any kind of possible statistics related to query, which can be used by optimizer to yield the even better plan compare to what it would have done otherwise.
Since most of the databases stores statistics for a particular column or relation but doesn't store statistics related to join of column or relation. Rather these databases just multiply the statistics of individual column/relation to get the statistics of join, which may not be always correct.

Example:
Lets say there is query as
SELECT * FROM EMPLOYEE WHERE GRADE>5 AND SALARY > 10000;

If we calculate independent stats for a and b.
suppose sel(GRADE) = .2 and sel(SALARY) = .2;

then sel (GRADE and SALARY) =
sel(GRADE) * sel (SALARY) = .04.
 
In all practical cases if we see, these two components will be highly dependent i.e. if first column satisfy,second column will also satisfy. Then in that case sel (GRADE and SALARY) should be .2 not .04. But current optimizer will be incorrect in this case and may give wrong plan.

Data Hint:

This kind of hints provides the information about the relationship/ dependency among relations or column to influence the plan instead of directly hinting to provide desired plan or direct selectivity value. Optimizer can consider dependency information to derive the actual selectivity.

Example:
Lets say there is a query as
SELECT * FROM TBL  WHERE ID1 = 5 AND ID2=NULL;
SELECT * FROM TBL  WHERE ID1 = 5 AND ID2!=NULL;

Now here if we specify that the dependency as
“If TBL.ID1 = 5 then TBL.ID2 is NULL”
then the optimizer will always consider this dependency pattern and accordingly combined statistics for these two columns can be choosen.

Note: This feature is not yet available in PG.

Conclusion:
Unlike other DB, we can provide some actual statistics information to optimizer to come out with the most optimal plan instead of directly telling planner to choose one specific plan.

Josh Berkus: South Bay PUG: Vitesse and Heikki at Adobe

From Planet PostgreSQL. Published on Apr 13, 2015.

The South Bay PUG is having a meetup on April 28th. Speakers will include CK Tan of PostgreSQL enhancement company Vitesse Data, and Heikki Linnakangas, PostgreSQL Committer. We do not expect to have live video for this meetup, sorry!

RSVP on the Meetup Page.

Umair Shahid: HOWTO create reports in Tableau with PostgreSQL database

From Planet PostgreSQL. Published on Apr 13, 2015.

For 2015, once again, Gartner’s Magic Quadrant for Business Intelligence and Analytics Platforms ranks Tableau pretty much at the top. How powerful would it be to combine Tableau with the world’s most advanced open source database, PostgreSQL, to create reports and analytics for businesses? This HOWTO walks you through the process of connecting Tableau with PostgreSQL and creating a simple report.

This HOWTO is based on the following configuration:

  • Windows 7
  • Tableau 9 Desktop
  • PostgreSQL 9.4.1

You can download the database used in this post from here: Database SQL (1 download ) .

Let’s create our first report!

Connecting to PostgreSQL

1) Launch Tableau. Click ‘More Servers …’ and select PostgreSQL from the menu as illustrated in the following screenshot:

 

2) Fill in the PostgreSQL connection properties and hit OK. If you don’t already have the required connection libraries installed, you will get an error as seen in following screenshot.

 

3) The (rather helpful) error dialog provides a link to download the libraries, click on it (requires working internet connection). This should take you to the driver’s section on Tableau’s official website. Locate PostgreSQL and download the corresponding setup file. See following screenshot for reference:

 

4) Run the downloaded file (may require administrator privileges). This will setup ODBC drivers and all system configurations required for PostgreSQL connectivity. Once you have completed setup, run Tableau Desktop again and connect to the PostgreSQL database you downloaded before the 1st step.

Creating a simple report

1) Once connected, you’ll find yourself on the Data Source tab. It lists the server, database and the tables as shown in the screen below.

 

2) You can drag and drop the tables you want to use. Alternatively, you can write a custom SQL for the required dataset. For the purpose of this blog, I have dragged and dropped ‘sales’ table. You can take a peek at a preview of the dataset in the result section of the window. Please note that you may have to click ‘Update’ in result pane to preview the result.

 

3) Click ‘Sheet’ tab (sheet2 in our example) to open the report sheet. The tab on the left will show 2 headers: Data and Measure. The former lists available dimensions whereas the latter lists all measurable values. Right pane in the window shows a pivot table like structure.

 

4) Now, let’s create a simple report that lists out sales and dates by country. To do this, simply drag ‘Sale’ measure and drop it on the data area in the pivot table. Now drag ‘Date’ and ‘Country’ from ‘Dimensions’ section and drop in ‘Rows’ area. And that’s it!

Refer to following screenshot for reference.

 

Adding extract criteria

1) Next, let’s try to filter the results by country. Start by dragging ‘Country’ from ‘Dimensions’ and dropping it in the ‘Filters’ area. In the dialog box that opens up, under the ‘General’ tab, click on ‘Select from list’. Next, click ‘All’ to select all the countries to show up and press OK.

 

2) Right-click ‘Country’ under ‘Dimensions’ and click Create -> Parameter.

 

3) The next dialog box specifies various properties for the parameter we are creating. Enter ‘Country Parameter‘ as Name, String as data type, and ‘List’ as ‘Allowable Values’. This last selection forces the user can to select a country name from the provided list. Click OK to confirm parameter properties.

 

4) ‘Country Parameter’ now appears under ‘Parameters’ section in the left pane.

 

5) Now open ‘Country’ filter properties and go to ‘Conditions’ tab. Select ‘By Formula’ and key in ‘[Country]=[Country Parameter]’ as shown in following screen shot. Click OK.

 

6) In order to present this option to the user, right click ‘County Parameter’ under ‘Parameters’ sub tab and select ‘Show Parameter Control’.

 

Running the report

Click ‘Presentation Mode’ OR press F-7 to preview. Select a country from ‘Country Parameter’ list to show its result.

 

Your first report with an extract criterion is complete!

Take a look at these freely available training videos from Tableau to learn more about what you can do using Tableau.

Have questions? Contact us NOW!

The post HOWTO create reports in Tableau with PostgreSQL database appeared first on Stormatics.

David E. Wheeler: PGXN Release Badges

From Planet PostgreSQL. Published on Apr 10, 2015.

Looks like it’s been close to two years since my last post on the PGXN blog. Apologies for that. I’ve thought for a while maybe I should organize an “extension of the week” series or something. Would there be interest in such a thing?

Meanwhile, I’m finally getting back to posting to report on a fun thing you can now do with your PGXN distributions. Thanks to the Version Badge service from the nice folks at Gemfury, you can badge your distributions! Badges look like this:

PGXN version

You’ve no doubt seem simlar badges for Ruby, Perl, and Python modules. Now the fun comes to PGXN. Want in? Assuming you have a disribution named pgfoo, just put code like this into the README file:

[![PGXN version](https://badge.fury.io/pg/pgfoo.svg)](https://badge.fury.io/pg/pgfoo)

This is Markdown format; use the syntax appropriate to your preferred README format to get the badg to show up on GitHub and PGXN.

That’s it! The badge will show the current releases version on PGXN, and the button will link through to PGXN.

Use Travis CI? You can badge your build status, too, as I’ve done for pgTAP, like this:

Build Status

[![Build Status](https://travis-ci.org/theory/pgtap.png)](https://travis-ci.org/theory/pgtap)

Coveralls provides patches, too. I’ve used them for Sqitch, though I’ve not yet taken the time figure out how to do coverage testing with PostgreSQL extensions. If you have, you can badge your current coverage like so:

Coverage Status

[![Coverage Status](https://coveralls.io/repos/theory/sqitch/badge.svg)](https://coveralls.io/r/theory/sqitch)

So get badging, and show off your PGXN distributions GitHub and elsewhere!

Shaun M. Thomas: PG Phriday: Functions and Addressing JSON Data

From Planet PostgreSQL. Published on Apr 10, 2015.

Fairly recently, a friend of mine presented a problem he wanted to solve with some JSON he had in a table. After he presented the end result he was trying to reach, I made the assumption that this would be pretty easy to do. But then I looked at the JSON Functions to try and find that quick fix. Though I read extensively and used rather liberal interpretations of the functions, there’s no way to directly manipulate JSON object contents with PostgreSQL.

Wait! Before you start yelling at me for being an idiot, I know what you’re going to say. I thought the same thing… at first. Go ahead and look, though. As of PostgreSQL 9.4, there is no built-in functionality to add or remove JSON elements without one or more intermediate transformation steps through PostgreSQL arrays or records. But that isn’t necessarily unexpected. Why?

Because PostgreSQL is a database. Its primary purpose is to store data and subsequently extract and view it. From this perspective, there’s no reason for PostgreSQL to have an entire library of JSON modification functions or operators. Regardless of this however, actions such as data merges and bulk updates still need to be possible. Yet all other fields allow a single update statement to append information, or otherwise perform a native calculation to replace the value in-line. There must be a way to do this with JSON too, without jumping through too many burning hoops.

Luckily there is, but it does require some preliminary work. Let’s start with a simple JSON document, as seen by PostgreSQL:

SELECT '{"Hairy": true, "Smelly": false}'::JSON;

               json
----------------------------------
 {"Hairy": true, "Smelly": false}

Ok. Now, how would I add an attribute named “Wobbly”? Well, I could pull the data into an external application, add it, and store the result. But suppose this was in a table of millions of records? That’s probably the least efficient way to modify them. This could be parallelized to a certain extent, but that requires a lot of scaffolding code and is way too much work for something so simple.

Instead, let’s create a function to do it for us. We’ve already established that PostgreSQL JSON manipulation is extremely limited, so what other option do we have? Here’s a python function:

CREATE or REPLACE FUNCTION json_update(data JSON, key TEXT, value JSON)
RETURNS JSON
AS $$

    if not key:
        return data

    from json import loads, dumps
    js = loads(data)
    js[key] = loads(value)
    return dumps(js)

$$ LANGUAGE plpythonu;

Now we could add the field with ease:

SELECT json_update('{"Hairy": true, "Smelly": false}'::JSON,
       'Wobbly', 'false'::JSON);

                    json_update                    
---------------------------------------------------
 {"Hairy": true, "Smelly": false, "Wobbly": false}

And if we want to get really fancy, there’s always PLV8:

CREATE or REPLACE FUNCTION json_update(data JSON, key TEXT, value JSON)
RETURNS JSON
AS $$
    if (key)
        data[key] = value;
    return data;

$$ LANGUAGE plv8;

SELECT json_update('{"Hairy": true, "Smelly": false}'::JSON,
       'Wobbly', 'false'::JSON);

                 json_update                  
----------------------------------------------
 {"Hairy":true,"Smelly":false,"Wobbly":false}

Though with PLV8, there are a couple of relatively minor caveats.

  1. PLV8 doesn’t work with JSONB yet, which is why all of these examples are in JSON instead.
  2. You might notice that it stripped all the extraneous whitespace, which may not be desirable.

Either way, both of these variants do something that PostgreSQL can’t do on its own. This is one of the reasons PostgreSQL is my favorite database; it’s so easy to extend and enhance.

Just as a thought experiment, which of these functional variants is faster? I didn’t use the IMMUTABLE or STRICT decorators, so it would be easy to run a loop of a few thousand iterations and see what the final run-time is. Here’s a modification of the test query:

EXPLAIN ANALYZE
SELECT json_update('{"Hairy": true, "Smelly": false}'::JSON,
       'Wobbly', 'false'::JSON)
  FROM generate_series(1, 100000);

On my test VM, the python function took around four seconds, while the PLV8 version only needed a second and a half. Clearly PLV8’s native handling of its own datatype helps here, and python having to repeatedly import the json library hurts its own execution. By adding IMMUTABLE, both fly through all 100-thousand iterations in less than 200ms.

Don’t be afraid to stray from SQL when using PostgreSQL. In fact, this might be a good case for thinking about PostgreSQL in an entirely different light. I might start calling it PGDB from now on, simply to honor its roots and its primary functionality. SQL is no longer the Alpha and Omega when it comes to its capabilities these days. So I feel it’s only right to adapt along with it.

Here’s to the future of PGDB!

gabrielle roth: PDXPUG: April meeting next week

From Planet PostgreSQL. Published on Apr 09, 2015.

When: 6-8pm Thursday April 16, 2015
Where: Iovation
Who: Eric Hanson
What: Aquameta release!

Eric Hanson will give a tutorial for how to build applications with Aquameta, an open source web development platform built entirely in PostgreSQL. Aquameta is about to be launched as open source, so we’ll do a quick launch recap, and then dive into the tutorial.


Our meeting will be held at Iovation, on the 32nd floor of the US Bancorp Tower at 111 SW 5th (5th & Oak). It’s right on the Green & Yellow Max lines. Underground bike parking is available in the parking garage; outdoors all around the block in the usual spots. No bikes in the office, sorry!

Elevators open at 5:45 and building security closes access to the floor at 6:30.

The building is on the Green & Yellow Max lines. Underground bike parking is available in the parking garage; outdoors all around the block in the usual spots.

See you there!


Project Repositories Have Moved to GitHub

By Matthew Flanagan from Django community aggregator: Community blog posts. Published on Apr 08, 2015.

With Google announcing last month that they are shutting down Google Code I have moved my old and unloved code repositories to GitHub. The main code base that seems to still be in use by others is the Django Full Serializer which I may split out into its own repository one day.

Saving Data from Strava

By GoDjango - Django Screencasts from Django community aggregator: Community blog posts. Published on Apr 08, 2015.

Getting data can be easy. Effectively saving it from one source to your database can be a pain. In this video we explore a way to get data from Strava and save it to our database with as little future trouble as possible.
Watch Now...

How-to detect and prevent malware infected user uploads

By Cloudinary Blog - Django from Django community aggregator: Community blog posts. Published on Apr 08, 2015.

Anti-malware icon Social networking sites allow users to upload images or other types of files that are immediately available to other users via news feeds or notifications. In some cases, attackers can directly spread infected files, but more commonly, they leverage the viral effect and the fact that users are simply unaware that their files are infected through sharing and collaborating with others. As a site owner or application developer, it is your responsibility to protect users and prevent these situations from occurring. Fortunately, Cloudinary makes this easier with its Metascan add-on.

Cloudinary's image management allows you to upload images, along with other files like PDFs, directly from your users’ browsers or mobile applications. While most modern image formats are fairly safe, nothing is guaranteed, and the various other file formats that users use to upload and share may be infected with malicious code. We want to ensure that you and your users are safe so that there's no chance of malware creeping into uploaded files. With Cloudinary's Metascan add-on, you can detect and prevent advanced known and unknown malware and threats by scanning and sanitizing your users' uploaded files.

Metascan anti-malware add-on

Brought to you by OPSWAT, Metascan is fully integrated into Cloudinary’s image management and manipulation service. While uploading images to the cloud using Cloudinary’s upload API, you can set the moderation upload API parameter to metascan in order to request to have your uploaded images and other files automatically scanned and removed if malware is detected. It’s also possible to select batches of previously uploaded files to scan using Cloudinary’s API.

With the Metascan add-on, you don't need to install any anti-malware software or build a file scanning pipeline. By simply using Cloudinary's image upload API, or Admin API for previously uploaded images, all of your images can be scanned and moderated accordingly.

Below is a sample code that requests a Metascan scan of a PDF document that is being uploaded to Cloudinary.

Ruby:
Cloudinary::Uploader.upload("local_file.pdf", 
  :moderation => "metascan")
PHP:
\Cloudinary\Uploader::upload("local_file.pdf", 
  array("moderation" => "metascan"));
Python:
cloudinary.uploader.upload("local_file.pdf",
  moderation = "metascan")
Node.js:
cloudinary.uploader.upload("local_file.pdf", 
  function(result) { console.log(result); }, 
  { moderation: "metascan" });
Java:
cloudinary.uploader().upload("local_file.pdf", 
  ObjectUtils.asMap("moderation", "metascan"));

With this request, Metascan performs scans with anti-malware engines (ESet, AVG, ClaimWin and Norman) on images and files that are being uploaded. Scanning is done asynchronously within seconds, meaning there is no impact on user experience. If Metascan detects malware in a file, the file is removed and the CDN cache is invalidated. The original file is stored in secondary storage, which enables you to recover it in case you want to override the Metascan results.

Additionally, you can set up notifications that let you know which files were marked as safe or rejected. You have the option of scanning your user uploaded files with Cloudinary before publishing them, allowing you to share only uploaded content that you know has gotten the green light.

As you can see in the screenshot below, Cloudinary’s online media library allows you to further control this process with multiple lists of pending scan requests, and approved or rejected files, permitting you to override these results if necessary. These can also be performed by the API. For example, if a user reports a file that wasn’t initially found to be malicious, you can mark it as problematic with a single click or API call and remove it or approve it if you think that it was mistakenly marked as containing malware. See our documentation to learn more about the Metascan add-on.

Metascan moderation queue

Final Notes

With all of the cyber threats in the world today, sharing on social networks makes malware distribution fast and dangerous. While most image formats are relatively safe, other formats, such as PDFs, are more risky. At Cloudinary, we are committed to helping you protect your users, website or web application from malware and viruses. Using Cloudinary’s Metscan add-on prevents vulnerabilities and enhances overall site security.

Metascan add-on screenshot

You can try the Metascan add-on for free, just by subscribing to the free add-on plan. If you don't have a Cloudinary account yet, sign up for a free account here.

strftime for datetime before 1900 year

By Andrey Zhukov's blog from Django community aggregator: Community blog posts. Published on Apr 08, 2015.

Recently I’ve got an error for birthdays before 1900 year

1
ValueError: year=1890 is before 1900; the datetime strftime() methods require year >= 1900

for this code:

1
2
import datetime
datetime.date(1890,7,1).strftime('%d.%m.%Y')

And it’s described in the documentation

The full set of format codes supported varies across platforms, because Python calls the platform C library’s strftime() function, and platform variations are common. To see the full set of format codes supported on your platform, consult the strftime(3) documentation.
The exact range of years for which strftime() works also varies across platforms. Regardless of platform, years before 1900 cannot be used.

One of the ways to solve this:

1
2
birthday = datetime.date(1890,7,1)
'{0.day:02d}.{0.month:02d}.{0.year:4d}'.format(birthday)

Andrew Dunstan: Dynamically disabling triggers without locks

From Planet PostgreSQL. Published on Apr 07, 2015.

Recently Simon Riggs committed a patch by himself and Andreas Karlsson to reduce the lock strength required by certain ALTER TABLE commands, including those to enable or disable triggers. Now the lock level required is SHARE ROW EXCLUSIVE instead of ACCESS EXCLUSIVE. That means it doesn't block SELECT commands any more, and isn't blocked by them, although it will still block and be blocked by INSERT, UPDATE and DELETE operations. Very nice.

However, without formally disabling a trigger you can tell it dynamically not to do anything in the current session without taking any locks at all. Here's a little bit of PLpgsql code I wrote recently for this sort of operation in an INSERT trigger:
    begin
disabled := current_setting('mypackage.foo_trigger_disabled');
exception
when others then disabled := 'false';
end;
if disabled = 'true' then
return NEW;
end if;
Note that this will only block the trigger from doing anything in sessions where this variable is set. But that's often exactly what you want. In the case this was written for, the trigger is redundant (and expensive) for certain bulk operations, but required for normal operations.  So in a session where we are performing the bulk operation, we can simply set this and avoid taking out a heavy lock on the table, and do this instead, before running our bulk operation:
    set mypackage.foo_trigger_disabled = 'true';
The code above is a bit ugly because of the requirement for the exception handler. There's a cure for that coming, too. David Christensen has submitted a patch to provide a form of current_setting() which will return NULL for unset variables instead of raising an exception.

Note, too, that you could use a value in a one-row one-column table if you wanted something that could apply in all sessions, not just the current session. It would be a bit less efficient, though. This mechanism is pretty light-weight.

Umair Shahid: HOWTO use JSON functionality in PostgreSQL

From Planet PostgreSQL. Published on Apr 07, 2015.

In a previous post, I talked about the excitement that surrounds NoSQL support in PostgreSQL. Today, I will dive a little deeper into the technical details of native JSON support that makes NoSQL possible.

Below, I explain how to use some basic functions and operators.

Creating a table with JSON column

So this is fairly simple. You can declare a JSON column just like a column of any other data type. Below, I create a table ‘sales’ with 2 columns, ‘id’ and ‘sale’, with the latter being a JSON.

json_sample=# CREATE TABLE sales (id INT, sale JSON);
CREATE TABLE

Inserting JSON data

Insertion of data is pretty close to that of any other data type as well, except that you have to make sure the data is in a valid JSON format. Below, I am inserting 4 records into the table, each with a JSON containing nodes for ‘customer_name’ and a nested JSON for ‘items’ containing a ‘description’ and purchased ‘quantity’.
John bought 4 cartons of milk:

json_sample=# INSERT INTO sales 
                   VALUES (1,'{ "customer_name": "John", "items": { "description": "milk", "quantity": 4 } }');
INSERT 0 1

Susan bought 2 loaves of bread:

json_sample=# INSERT INTO sales 
                   VALUES (2,'{ "customer_name": "Susan", "items": { "description": "bread", "quantity": 2 } }');
INSERT 0 1

Mark bought a dozen bananas:

json_sample=# INSERT INTO sales 
                   VALUES (3,'{ "customer_name": "Mark", "items": { "description": "bananas", "quantity": 12 } }');
INSERT 0 1

Jane bought a pack of cereal:

json_sample=# INSERT INTO sales 
                   VALUES (4,'{ "customer_name": "Jane", "items": { "description": "cereal", "quantity": 1 } }');
INSERT 0 1

Retrieving JSON data

Like insertion, retrieving the JSON formatted data is fairly straightforward as well. Below, I am retrieving the data I inserted in the previous section.

json_sample=# SELECT * FROM sales;
id |                                      sale
----+------------------------------------------------------------------------------------
1 | { "customer_name": "John", "items": { "description": "milk", "quantity": 4 } }
2 | { "customer_name": "Susan", "items": { "description": "bread", "quantity": 2 } }
3 | { "customer_name": "Mark", "items": { "description": "bananas", "quantity": 12 } }
4 | { "customer_name": "Jane", "items": { "description": "cereal", "quantity": 1 }

Retrieving JSONs – The ‘->’ and ‘->>’ operators

Now comes the real fun part! PostgreSQL provides native operators to retrieve individual nodes of the JSON object … very powerful indeed. In this section, I discuss the ‘->’ operator, which return a JSON object and the ‘->>’ operator, which returns TEXT.

Retrieving as a JSON:

json_sample=# SELECT sale->'customer_name' AS name FROM sales;
name
---------
"John"
"Susan"
"Mark"
"Jane"
(4 rows)

Retrieving as TEXT:

json_sample=# SELECT sale->>'customer_name' AS name FROM sales;
name
-------
John
Susan
Mark
Jane
(4 rows)

Chaining the ‘->’ and ‘->>’ operators

Since ‘->’ returns a JSON object, you can use it to return a nested object within the data and chain it with the operator ‘->>’ to retrieve a specific node.

json_sample=# SELECT id, sale->'items'->>'quantity' AS quantity FROM sales;
id | quantity
----+----------
1 | 4
2 | 2
3 | 12
4 | 1
(4 rows)

Using JSONs in extract criteria for queries

The operators discussed in the previous section can be used in the WHERE clause of a query to specify an extract criteria. A few examples, using the same data set, below.

Searching for a specific description of an item within a sale:

json_sample=# SELECT * FROM sales WHERE sale->'items'->>'description' = 'milk';
id |                                     sale
----+--------------------------------------------------------------------------------
1 | { "customer_name": "John", "items": { "description": "milk", "quantity": 4 } }
(1 row)

Searching for a specific quantity as TEXT:

json_sample=# SELECT * FROM sales WHERE sale->'items'->>'quantity' = 12::TEXT;
id |                                       sale
----+------------------------------------------------------------------------------------
3 | { "customer_name": "Mark", "items": { "description": "bananas", "quantity": 12 } }
(1 row)

Searching for a specific quantity as INTEGER:

json_sample=# SELECT * FROM sales WHERE CAST(sale->'items'->>'quantity' AS integer)  = 2;
 id |                                       sale                                       
----+----------------------------------------------------------------------------------
  2 | { "customer_name": "Susan", "items": { "description": "bread", "quantity": 2 } }
(1 row)

Using JSON nodes in aggregate functions

Once you understand how to retrieve individual nodes of a JSON object, you can easily use the retrieved values in aggregate functions as well.

json_sample=# SELECT SUM(CAST(sale->'items'->>'quantity' AS integer)) AS total_quantity_sold FROM sales;
total_quantity_sold
---------------------
19
(1 row)

JSON functions in PostgreSQL

Let’s go through some functions that PostgreSQL provides for manipulating JSON objects.

json_each

This function expands the outermost JSON object into a set of key/value pairs. Notice that the nested JSONs are not expanded.

json_sample=# SELECT json_each(sale) FROM sales;
json_each
--------------------------------------------------------------
(customer_name,"""John""")
(items,"{ ""description"": ""milk"", ""quantity"": 4 }")
(customer_name,"""Susan""")
(items,"{ ""description"": ""bread"", ""quantity"": 2 }")
(customer_name,"""Mark""")
(items,"{ ""description"": ""bananas"", ""quantity"": 12 }")
(customer_name,"""Jane""")
(items,"{ ""description"": ""cereal"", ""quantity"": 1 }")
(8 rows)

json_object_keys

Returns set of keys in the outermost JSON object. Again, notice that the nested keys are not displayed.

json_sample=# SELECT json_object_keys(sale) FROM sales;
json_object_keys
------------------
customer_name
items
customer_name
items
customer_name
items
customer_name
items
(8 rows)

json_typeof

Returns the type of the outermost JSON value as a text string. Possible types are ‘object’, ‘array’, ‘string’, ‘number’, ‘boolean’, and NULL.

json_sample=# SELECT json_typeof(sale->'items'), json_typeof(sale->'items'->'quantity') FROM sales;
json_typeof | json_typeof
-------------+-------------
object     | number
object     | number
object     | number
object     | number
(4 rows)

json_object

Builds a JSON object out of a text array. The function can be used in one of two ways:

(1) Array with exactly one dimension with an even number of members. In this case the elements are taken as alternating key/value pairs.

json_sample=# SELECT json_object('{key1, 6.4, key2, 9, key3, "value"}');
json_object
--------------------------------------------------
{"key1" : "6.4", "key2" : "9", "key3" : "value"}
(1 row)

(2) Array with two dimensions such that each inner array has exactly two elements. In this case, the inner array elements are taken as a key/value pair.

json_sample=# SELECT * FROM json_object('{{key1, 6.4}, {key2, 9}, {key3, "value"}}');
json_object
--------------------------------------------------
{"key1" : "6.4", "key2" : "9", "key3" : "value"}
(1 row)

to_json

Returns the value as a JSON object.

json_sample=# CREATE TABLE json_test (id INT, name TEXT);
CREATE TABLE
json_sample=# INSERT INTO json_test VALUES (1, 'Jack');
INSERT 0 1
json_sample=# INSERT INTO json_test VALUES (2, 'Tom');
INSERT 0 1
json_sample=# SELECT row_to_json(row(id, name)) FROM json_test;
row_to_json
----------------------
{"f1":1,"f2":"Jack"}
{"f1":2,"f2":"Tom"}
(2 rows)

 

PostgreSQL official documentation has a more comprehensive listing of the available JSON functions.

Have questions? Contact us NOW!

The post HOWTO use JSON functionality in PostgreSQL appeared first on Stormatics.

Hubert 'depesz' Lubaczewski: Waiting for 9.5 – psql: add asciidoc output format

From Planet PostgreSQL. Published on Apr 07, 2015.

On 31st of March, Bruce Momjian committed patch: psql: add asciidoc output format   Patch by Szymon Guz, adjustments by me   Testing by Michael Paquier, Pavel Stehule To be honest, when Szymon posted first mail about asciidoc – it was the first time I heard about it. Immediately I thought: “why not markdown? or […]

Andrew Dunstan: Fun with Raspberry Pi 2 and the buildfarm

From Planet PostgreSQL. Published on Apr 06, 2015.

Here's a picture of my two Raspberry Pi 2 boxes, both running headless and wireless.


One is running Raspbian, installed via NOOBS, and the other Fidora, a remix of Fedora 21 for Raspberry Pi 2. It turned out that Pidora doesn't work on the Raspberry Pi 2, a fact that is extremely well hidden on the Raspberry Pi web site.

I have set up test buildfarm animals on both of these. But something odd is happening. They are both getting intermittent failures of the stats regression test. Sometimes it happens during "make check", sometimes during "make installcheck" and sometimes during testing of pg_upgrade (which in turn runs "make installcheck").

These machines are not anything like speed demons. Far from it. But we also run other slow machines without getting this happening all the time. So I'm a bit perplexed about what might be going on.

Incidentally, if you want to play with one of these, I do recommend getting a starter kit from Amazon or elsewhere. It's probably cheaper than buying everything separately, and gets you everything you need to get started. Well worth the $69.99.

David Christensen: PgConf 2015 NYC Recap

From Planet PostgreSQL. Published on Apr 06, 2015.

I recently just got back from PGConf 2015 NYC.  It was an invigorating, fun experience, both attending and speaking at the conference.

What follows is a brief summary of some of the talks I saw, as well as some insights/thoughts:

On Thursday:

"Managing PostgreSQL with Puppet" by Chris Everest.  This talk covered experiences by CoverMyMeds.com staff in deploying PostgreSQL instances and integrating with custom Puppet recipes.

"A TARDIS for your ORM - application level timetravel in PostgreSQL" by Magnus Hagander. Demonstrated how to construct a mirror schema of an existing database and manage (via triggers) a view of how data existed at some specific point in time.  This system utilized range types with exclusion constraints, views, and session variables to generate a similar-structured schema to be consumed by an existing ORM application.

"Building a 'Database of Things' with Foreign Data Wrappers" by Rick Otten.  This was a live demonstration of building a custom foreign data wrapper to control such attributes as hue, brightness, and on/off state of Philips Hue bulbs.  Very interesting live demo, nice audience response to the control systems.  Used a python framework to stub out the interface with the foreign data wrapper and integrate fully.

"Advanced use of pg_stat_statements: Filtering, Regression Testing & More" by Lukas Fittl.  Covered how to use the pg_stat_statements extension to normalize queries and locate common performance statistics for the same query.  This talk also covered the pg_query tool/library, a Ruby tool to parse/analyze queries offline and generate a JSON object representing the query.  The talk also covered the example of using a test database and the pg_stat_statements views/data to perform query analysis to theorize about planning of specific queries without particular database indexes, etc.

On Friday:

"Webscale's dead! Long live Postgres!" by Joshua Drake.  This talk covered improvements that PostgreSQL has made over the years, specific technologies that they have incorporated such as JSON, and was a general cheerleading effort about just how awesome PostgreSQL is.  (Which of course we all knew already.)  The highlight of the talk for me was when JD handed out "prizes" at the end for knowing various factoids; I ended up winning a bottle of Macallan 15 for knowing the name of the recent departing member of One Direction.  (Hey, I have daughters, back off!)

"The Elephants In The Room: Limitations of the PostgreSQL Core Technology" by Robert Haas.  This was probably the most popular talk that I attended.  Robert is one of the core members of the PostgreSQL development team, and is heavily knowledgeable in the PostgreSQL internals, so his opinions of the existing weaknesses carry some weight.  This was an interesting look forward at possible future improvements and directions the PostgreSQL project may take.  In particular, Robert looked at the IO approach Postgres currently take and posits a Direct IO idea to give Postgres more direct control over its own IO scheduling, etc.  He also mentioned the on-disk format being somewhat suboptimal, Logical Replication as an area needing improvement, infrastructure needed for Horizontal Scalability and Parallel Query, and integrating Connection Pooling into the core Postgres product.

"PostgreSQL Performance Presentation (9.5devel edition)" by Simon Riggs.  This talked about some of the improvements in the 9.5 HEAD; in particular looking at the BRIN index type, an improvement in some cases over the standard btree index method.  Additional metrics were shown and tested as well, which demonstrated Postgres 9.5's additional performance improvements over the current version.

"Choosing a Logical Replication System" by David Christensen.  As the presenter of this talk, I was also naturally required to attend as well.  This talk covered some of the existing logical replication systems including Slony and Bucardo, and broke down situations where each has strengths.

"The future of PostgreSQL Multi-Master Replication" by Andres Freund.  This talk primarily covered the upcoming BDR system, as well as the specific infrastructure changes in PostgreSQL needed to support these features, such as logical log streaming.  It also looked at the performance characteristics of this system.  The talk also wins for the most quote-able line of the conference:  "BDR is spooning Postgres, not forking", referring to the BDR project's commitment to maintaining the code in conjunction with core Postgres and gradually incorporating this into core.

As part of the closing ceremony, there were lightning talks as well; quick-paced talks (maximum of 5 minutes) which covered a variety of interesting, fun and sometimes silly topics.  In particular some memorable ones were one about using Postgres/PostGIS to extract data about various ice cream-related check-ins on Foursquare, as well as one which proposed a generic (albeit impractical) way to search across all text fields in a database of unknown schema to find instances of key data.

As always, it was good to participate in the PostgreSQL community, and look forward to seeing participants again at future conferences.

Hubert 'depesz' Lubaczewski: Waiting for 9.5 – Add stats for min, max, mean, stddev times to pg_stat_statements.

From Planet PostgreSQL. Published on Apr 06, 2015.

On 27th of March, Andrew Dunstan committed patch: Add stats for min, max, mean, stddev times to pg_stat_statements.   The new fields are min_time, max_time, mean_time and stddev_time.   Based on an original patch from Mitsumasa KONDO, modified by me. Reviewed by Petr Jelínek. While pg_stat_statements provides a lot of information about statements, it's timing […]

Paul Ramsey: PostGIS 2.0.7 & 2.1.7 Released

From Planet PostgreSQL. Published on Apr 05, 2015.

Due to a critical bug in GeoJSON ingestion we have made an early release of versions 2.0.7 and 2.1.7. If you are running an earlier version on a public site and accepting incoming GeoJSON, we recommend that you update as soon as possible.

http://download.osgeo.org/postgis/source/postgis-2.1.7.tar.gz

http://download.osgeo.org/postgis/source/postgis-2.0.7.tar.gz

View all closed tickets for 2.0.7.

Hubert 'depesz' Lubaczewski: Waiting for 9.5 – Add support for index-only scans in GiST.

From Planet PostgreSQL. Published on Apr 05, 2015.

On 26th of March, Heikki Linnakangas committed patch: Add support for index-only scans in GiST.   This adds a new GiST opclass method, 'fetch', which is used to reconstruct the original Datum from the value stored in the index. Also, the 'canreturn' index AM interface function gains a new 'attno' argument. That makes it possible […]

Peter Geoghegan: Abbreviated keys for numeric to accelerate numeric sorts

From Planet PostgreSQL. Published on Apr 04, 2015.

Andrew Gierth's numeric abbreviated keys patch was committed recently. This commit added abbreviation/sortsupport for the numeric type (the PostgreSQL type which allows practically arbitrary precision, typically recommended for representing monetary values).

The encoding scheme that Andrew came up with is rather clever - it has an excellent tendency to concentrate entropy from the original values into the generated abbreviated keys in real world cases. As far as accelerating sorts goes, numeric abbreviation is at least as effective as the original text abbreviation scheme. I easily saw improvements of 6x-7x with representative queries that did not spill to disk (i.e. that used quicksort). In essence, the patch makes sorting numeric values almost as cheap as sorting simple integers, since that is often all that is actually required during sorting proper (the abbreviated keys compare as integers, except that the comparison is inverted to comport with how abbreviation builds abbreviated values from numerics as tuples are copied into local memory ahead of sorting - see the patch for exact details).

Separately, over lunch at pgConf.US in New York, Corey Huinker complained about a slow, routine data warehousing CREATE INDEX operation that took far too long. The indexes in question were built on a single text column. I suggested that Corey check out how PostgreSQL 9.5 performs, where this operation is accelerated by text abbreviation, often very effectively.

Corey chose an organic set of data that could be taken as a reasonable proxy for how PostgreSQL behaves when he performs these routine index builds. In all cases maintenance_work_mem was set to 64MB, meaning that an external tapesort is always required - those details were consistent. This was a table with 1.8 million rows. Apparently, on PostgreSQL 9.4, without abbreviation, the CREATE INDEX took 10 minutes and 19 seconds in total. On PostgreSQL 9.5, with identical settings, it took only 51.3 seconds - a 12x improvement! This was a low cardinality pre-sorted column, but if anything that is a less compelling case for abbreviation - I think that the improvements could sometimes be even greater when using external sorts on big servers with fast CPUs. Further organic benchmarks of abbreviated key sorts are very welcome. Of course, there is every reason to imagine that abbreviation would now improve things just as much if not more with large numeric sorts that spill to disk.

Future work

With numeric abbreviation committed, and support for the "datum" case likely to be committed soon, you might assume that abbreviation as a topic on the pgsql-hackers development mailing list had more or less played out (the "datum " sort case is used by things like "SELECT COUNT(DISTINCT FOO) ..." - this is Andrew Gierth's work again).  You might now reasonably surmise that it would be nice to have support for the default B-Tree opclasses of one or two other types, like character(n), but that's about it, since clearly abbreviation isn't much use for complex/composite types - we're almost out of interesting types to abbreviate. However, I think that work on abbreviated keys is far from over. Abbreviation as a project is only more or less complete as a technique to accelerate sorting, but that's likely to only be half the story (Sorry Robert!).

I intend to undertake research on using abbreviated keys within internal B-Tree pages in the next release cycle. Apart from amortizing the cost of comparisons that are required to service index scans, I suspect that they can greatly reduce the number of cache misses by storing abbreviated keys inline in the ItemId array of internal B-Tree pages. Watch this space!


Michael Paquier: Postgres 9.5 feature highlight: Default shutdown mode of pg_ctl to fast

From Planet PostgreSQL. Published on Apr 04, 2015.

This week, I wanted to share something that may impact many users of Postgres, with this commit changing a behavior that had for a long time the same default:

commit: 0badb069bc9f590dbc1306ccbd51e99ed81f228c
author: Bruce Momjian <bruce@momjian.us>
date: Tue, 31 Mar 2015 11:46:27 -0400
pg_ctl:  change default shutdown mode from 'smart' to 'fast'

Retain the order of the options in the documentation.

pg_ctl has three shutdown modes:

  • smart, the polite one, waits patiently for all the active clients connections to be disconnected before shutting down the server. This is the default mode for Postgres for ages.
  • immediate, the brute-force one, aborts all the server processes without thinking, leading to crash recovery when the instance is restarted the next time.
  • fast, takes an intermediate approach by rollbacking all the existing connections and then shutting down the server.

Simply, the "smart" mode has been considered the default because it is the least distuptive, particularly it will wait for a backup to finish before shutting down the server. It has been (justly) discussed that it was not enough aggresive, users being sometimes surprised that a shutdown requested can finish with a timeout because a connection has been for example left open, hence the default has been switched to "fast".

This is not complicated litterature, however be careful if you had scripts that relied on the default behavior of pg_ctl when switching to 9.5, particularly for online backups that will be immediately terminated at shutdown with the new default.

Hubert 'depesz' Lubaczewski: Waiting for 9.5 – Add pg_rewind, for re-synchronizing a master server after failback.

From Planet PostgreSQL. Published on Apr 04, 2015.

On 23rd of March, Heikki Linnakangas committed patch: Add pg_rewind, for re-synchronizing a master server after failback.   Earlier versions of this tool were available (and still are) on github.   Thanks to Michael Paquier, Alvaro Herrera, Peter Eisentraut, Amit Kapila, and Satoshi Nagayasu for review. So, we have a situation, where we have master […]

Glyn Astill: Being cavalier with slony nodes and pg_dump/pg_restore

From Planet PostgreSQL. Published on Apr 02, 2015.

It’s generally a bad idea to do logical dump/restores of slony nodes, and for this reason slony provides the CLONE PREPARE/CLONE FINISH action commands to clone subscriber nodes. In this instance though, I’ve a test environment where I’d stopped the slons, dumped out and dropped a subscriber database and then gone on to do some […]

Powerful image manipulation and categorization with facial attribute detection

By Cloudinary Blog - Django from Django community aggregator: Community blog posts. Published on Apr 02, 2015.

Face attributes

Face Detection is a great feature that enables the automatic modification of images according to the detected faces within an image, making it simple to intelligently crop, position, resize and transform your images appropriately.

Facial Attribute Detection takes the process even further and extracts meaningful advanced data about the face(s) in the image, including the exact location of facial features. This allows you even greater control over your image categorization, and to automatically use these details to smartly crop, position, rotate and overlay images based on the detected facial features.

Facial Attribute Detection lets you know more than just the location of a person's facial features. How are they posed in 3 dimensions? Is the person wearing glasses? Do they have their eyes closed? Mouth open? Have a mustache or beard? What is the person's race, age and gender? What emotion are they displaying? Are they smiling? How beautiful are they? Retrieving this information makes it a simple matter to automatically categorize and tag your collection of images.

All of this is made possible with the ReKognition add-on, which has been directly integrated within Cloudinary’s infrastructure, further extending Cloudinary’s built-in face detection to a robust Facial Attribute Detection feature. By simply setting the detection parameter to rekognition_face when calling Cloudinary's upload API, ReKognition is utilized to automatically extract detailed face attributes from the uploaded image.

Ruby:
Cloudinary::Uploader.upload("woman.jpg", 
              :detection => "rekognition_face")
PHP:
\Cloudinary\Uploader::upload("woman.jpg", 
              array(
               "detection" => "rekognition_face"));
Python:
cloudinary.uploader.upload("woman.jpg", 
              detection = "rekognition_face")
Node.js:
cloudinary.uploader.upload("woman.jpg", 
              function(result) {console.log(result); }, { detection: "rekognition_face" });
Java:
cloudinary.uploader().upload("woman.jpg", 
              Cloudinary.asMap("detection", "rekognition_face"));

Original woman image

The example JSON snippet displayed from the example image above contains the result returned from the face ReKognition request, which includes very detailed information regarding the face that was automatically detected in the image.

{"rekognition_face": 
 "status": "complete",
  "data": [
    {
      "boundingbox": {
        "tl": {"x": 231.45, "y": 102.52},
        "size": {"width": 240.77, "height": 240.77 }},
      "confidence": 1,
      "eye_left": {"x":309.6, "y": 190.1},
      "eye_right": {"x": 407.9, "y": 213.6},
      "nose": {"x": 199.1, "y": 204.0},
      
      
      
      "smile": 0.96,
      "glasses": 0.01,
      "sunglasses": 0.04,
      "beard": 0,
      "mustache": 0,
      "eye_closed": 0,
      "mouth_open_wide": 0.73,
      "beauty": 0.63531,
      "sex": 1
    }
  ]
}

You can also use Cloudinary's Admin API to apply ReKognition face detection to already uploaded images (based on their public IDs), and the face attributes that were previously extracted are also available using the Admin API's show resource details method (see the Documentation for more details on how to accomplish this).

Face detection based cropping

Based on the position of facial attributes detected by the ReKognition add-on, Cloudinary can crop your images to focus on the detected facial features, while providing a large set of image transformation and cropping options when using a Cloudinary delivery URL or calling Cloudinary's image API.

To focus an automatic crop on the detected faces, simply set the crop parameter to thumb, fill or crop and the gravity parameter to rek_faces (set gravity to rek_face for focusing on the single largest detected face in the image). The resulting images are dynamically generated on-the-fly and the result is delivered via a fast CDN.

Original photo

The following code sample generates a 150x150 thumbnail of the nice_coupleimage shown above, using multiple face detection based cropping.

Ruby:
cl_image_tag("nice_couple.jpg", :width=>150, :height=>150, :crop=>:thumb, :gravity=>:rek_faces)
PHP:
cl_image_tag("nice_couple.jpg", array("width"=>150, "height"=>150, "crop"=>"thumb", "gravity"=>"rek_faces"))
Python:
CloudinaryImage("nice_couple.jpg").image(width=150, height=150, crop="thumb", gravity="rek_faces")
Node.js:
cloudinary.image("nice_couple.jpg", {width: 150, height: 150, crop: "thumb", gravity: "rek_faces"})
Java:
cloudinary.url().transformation(new Transformation().width(150).height(150).crop("thumb").gravity("rek_faces")).imageTag("nice_couple.jpg")
jQuery:
$.cloudinary.image("nice_couple.jpg", {width: 150, height: 150, crop: "thumb", gravity: "rek_faces"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(150).Height(150).Crop("thumb").Gravity("rek_faces")).BuildImageTag("nice_couple.jpg")
150x150 thumbnail of nice_couple.jpg

Eyes detection based cropping

Cloudinary can also dynamically crop your images based on the position of detected eyes. Simply set the gravity parameter to rek_eyes (g_rek_eyes for URLs) to center the image on the detected eyes. The example below delivers a 200x60 thumbnail centered on the eyes:

Ruby:
cl_image_tag("woman.jpg", :width=>200, :height=>60, :crop=>:thumb, :gravity=>:rek_eyes)
PHP:
cl_image_tag("woman.jpg", array("width"=>200, "height"=>60, "crop"=>"thumb", "gravity"=>"rek_eyes"))
Python:
CloudinaryImage("woman.jpg").image(width=200, height=60, crop="thumb", gravity="rek_eyes")
Node.js:
cloudinary.image("woman.jpg", {width: 200, height: 60, crop: "thumb", gravity: "rek_eyes"})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(60).crop("thumb").gravity("rek_eyes")).imageTag("woman.jpg")
jQuery:
$.cloudinary.image("woman.jpg", {width: 200, height: 60, crop: "thumb", gravity: "rek_eyes"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(60).Crop("thumb").Gravity("rek_eyes")).BuildImageTag("woman.jpg")
200x60 thumbnail centered on eyes

Facial overlays

Thanks to the detailed information on the position of facial attributes detected by ReKognition, Cloudinary can add overlays while taking into account the pose of the face, and automatically scale and rotate the overlay accordingly.

Ruby:
cl_image_tag("HarlequinMask.jpg")
PHP:
cl_image_tag("HarlequinMask.jpg")
Python:
CloudinaryImage("HarlequinMask.jpg").image()
Node.js:
cloudinary.image("HarlequinMask.jpg")
Java:
cloudinary.url().imageTag("HarlequinMask.jpg")
jQuery:
$.cloudinary.image("HarlequinMask.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("HarlequinMask.jpg")
Harlequin mask

For example, in order to automatically overlay the above image of a harlequin mask scaled to 160% relative to the detected eyes in the main image:

Ruby:
cl_image_tag("woman.jpg", :overlay=>"HarlequinMask", :width=>1.6, :crop=>:scale, :gravity=>:rek_eyes, :flags=>:region_relative)
PHP:
cl_image_tag("woman.jpg", array("overlay"=>"HarlequinMask", "width"=>1.6, "crop"=>"scale", "gravity"=>"rek_eyes", "flags"=>"region_relative"))
Python:
CloudinaryImage("woman.jpg").image(overlay="HarlequinMask", width=1.6, crop="scale", gravity="rek_eyes", flags="region_relative")
Node.js:
cloudinary.image("woman.jpg", {overlay: "HarlequinMask", width: 1.6, crop: "scale", gravity: "rek_eyes", flags: "region_relative"})
Java:
cloudinary.url().transformation(new Transformation().overlay("HarlequinMask").width(1.6).crop("scale").gravity("rek_eyes").flags("region_relative")).imageTag("woman.jpg")
jQuery:
$.cloudinary.image("woman.jpg", {overlay: "HarlequinMask", width: 1.6, crop: "scale", gravity: "rek_eyes", flags: "region_relative"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Overlay("HarlequinMask").Width(1.6).Crop("scale").Gravity("rek_eyes").Flags("region_relative")).BuildImageTag("woman.jpg")
Harlequin masked face

Heres another example, this time with glasses.

Ruby:
cl_image_tag("glasses.jpg")
PHP:
cl_image_tag("glasses.jpg")
Python:
CloudinaryImage("glasses.jpg").image()
Node.js:
cloudinary.image("glasses.jpg")
Java:
cloudinary.url().imageTag("glasses.jpg")
jQuery:
$.cloudinary.image("glasses.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("glasses.jpg")
Glasses

Overlaying the above image scaled to 150% relative to the detected eyes in the main image, which is then presented as a 200 pixel wide round thumbnail centered on the face:

Ruby:
cl_image_tag("woman.jpg", :transformation=>[
  {:overlay=>"glasses", :width=>1.5, :crop=>:scale, :gravity=>:rek_eyes, :flags=>:region_relative},
  {:radius=>"max", :width=>200, :crop=>:thumb, :gravity=>:face}
  ])
PHP:
cl_image_tag("woman.jpg", array("transformation"=>array(
  array("overlay"=>"glasses", "width"=>1.5, "crop"=>"scale", "gravity"=>"rek_eyes", "flags"=>"region_relative"),
  array("radius"=>"max", "width"=>200, "crop"=>"thumb", "gravity"=>"face")
  )))
Python:
CloudinaryImage("woman.jpg").image(transformation=[
  {"overlay": "glasses", "width": 1.5, "crop": "scale", "gravity": "rek_eyes", "flags": "region_relative"},
  {"radius": "max", "width": 200, "crop": "thumb", "gravity": "face"}
  ])
Node.js:
cloudinary.image("woman.jpg", {transformation: [
  {overlay: "glasses", width: 1.5, crop: "scale", gravity: "rek_eyes", flags: "region_relative"},
  {radius: "max", width: 200, crop: "thumb", gravity: "face"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .overlay("glasses").width(1.5).crop("scale").gravity("rek_eyes").flags("region_relative").chain()
  .radius("max").width(200).crop("thumb").gravity("face")).imageTag("woman.jpg")
jQuery:
$.cloudinary.image("woman.jpg", {transformation: [
  {overlay: "glasses", width: 1.5, crop: "scale", gravity: "rek_eyes", flags: "region_relative"},
  {radius: "max", width: 200, crop: "thumb", gravity: "face"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Overlay("glasses").Width(1.5).Crop("scale").Gravity("rek_eyes").Flags("region_relative").Chain()
  .Radius("max").Width(200).Crop("thumb").Gravity("face")).BuildImageTag("woman.jpg")
Glasses overlayed on eyes

Summary

The ReKognition add-on is utilized to automatically extract detailed face attributes from your images, and enables advanced image manipulation and categorization based on the detected facial data, with relative scaling and rotation of overlays achieved automatically.

ReKognition detect face attributes add-on

The ReKognition add-on is available to all our free and paid plans. If you don't have a Cloudinary account, you are welcome to sign up to our free account and try it out.

Hubert 'depesz' Lubaczewski: Waiting for 9.5 – Allow foreign tables to participate in inheritance. – A.K.A. PostgreSQL got sharding.

From Planet PostgreSQL. Published on Apr 02, 2015.

On 22nd of March, Tom Lane committed patch: Allow foreign tables to participate in inheritance.   Foreign tables can now be inheritance children, or parents. Much of the system was already ready for this, but we had to fix a few things of course, mostly in the area of planner and executor handling of row […]

DiamondHacks 2015 Recap

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Apr 02, 2015.

Image via Diamond Hacks Facebook Page

This past weekend, Technical Director Mark Lavin came out to support DiamondHacks, NCSU’s first ever hackathon and conference event for women interested in computer science. Not only is NCSU Mark’s alma mater, but he’s also a strong supporter of co-organizer Girl Develop It RDU (GDI), of which Caktus is an official sponsor.

The weekend’s events began Saturday with nervous excitement as Facebook developer Erin Summers took the stage for her keynote address. Most memorable for Mark was a moment towards the end of Summers’ talk, in which she called for collaboration between neighboring audience members. It was at this point Mark first realized he was the only male in the room—a unique experience for a male developer. “I’m sure there’s lots of women who have felt the way I did,” Mark commented. The moment not only flipped the norms of a traditionally male-dominated field, but also filled Mark with a renewed appreciation for the importance of active inclusivity in the tech industry.

Aside from helping fill swag bags for the weekend’s participants and attending several of the talks, Mark gave a lightning talk, “Python and Django: Web Development Batteries Included” on Python and Django. Knowing attendees would be thinking about their upcoming projects and which language to build in, Mark chose to advocate for Django (he’s a little biased as the co-author of Lightweight Django). He highlighted the overall uses of Python as well as the extensiveness of its standard library. According to Mark, “Python comes with a lot of built-in functionality,” so it’s a great coding language for beginning developers. Mark also covered the basic Django view and model in his talk, emphasizing the features that make Django a complete framework—an excellent language for a hackathon.

Since supporting diversity in the tech industry was a key focus of the day, Mark also wanted to emphasize the inclusiveness of the Python and Django communities. From the diversity statement on Python’s website, to Django’s code of conduct, the Python community and Django subcommunity have been at the forefront of advocating for diversity and inclusion in the tech world. For Mark, this element has and continues to be “important for the growth of [the] language,” and has contributed to the vitality of these communities.

All in all the weekend was a great success, with especially memorable talks given by speakers working for Google, Trinket, and Hirease. Mark was impressed with the students’ enthusiasm and focus and lingered after both of iterations his talk to speak with attendees about their careers and interests. The next day he was equally affected by the range and talent behind Sunday’s hackathon projects as he followed the progress of various teams on Twitter. “These are the students [who] are going to help define what’s next,” he remarked.

Can’t get enough of Python, Django, and the talented Mark Lavin? Neither can we. Mark will be leading a workshop at PyCon on Building SMS Applications with Django along with fellow Cakti David Ray and our code school’s lead instructor, Caleb Smith. We’ll hope to see you there!

DiamondHacks 2015 Recap

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Apr 02, 2015.

Image via Diamond Hacks Facebook Page

This past weekend, Technical Director Mark Lavin came out to support DiamondHacks, NCSU’s first ever hackathon and conference event for women interested in computer science. Not only is NCSU Mark’s alma mater, but he’s also a strong supporter of co-organizer Girl Develop It RDU (GDI), of which Caktus is an official sponsor.

The weekend’s events began Saturday with nervous excitement as Facebook developer Erin Summers took the stage for her keynote address. Most memorable for Mark was a moment towards the end of Summers’ talk, in which she called for collaboration between neighboring audience members. It was at this point Mark first realized he was the only male in the room—a unique experience for a male developer. “I’m sure there’s lots of women who have felt the way I did,” Mark commented. The moment not only flipped the norms of a traditionally male-dominated field, but also filled Mark with a renewed appreciation for the importance of active inclusivity in the tech industry.

Aside from helping fill swag bags for the weekend’s participants and attending several of the talks, Mark gave a lightning talk, “Python and Django: Web Development Batteries Included” on Python and Django. Knowing attendees would be thinking about their upcoming projects and which language to build in, Mark chose to advocate for Django (he’s a little biased as the co-author of Lightweight Django). He highlighted the overall uses of Python as well as the extensiveness of its standard library. According to Mark, “Python comes with a lot of built-in functionality,” so it’s a great coding language for beginning developers. Mark also covered the basic Django view and model in his talk, emphasizing the features that make Django a complete framework—an excellent language for a hackathon.

Since supporting diversity in the tech industry was a key focus of the day, Mark also wanted to emphasize the inclusiveness of the Python and Django communities. From the diversity statement on Python’s website, to Django’s code of conduct, the Python community and Django subcommunity have been at the forefront of advocating for diversity and inclusion in the tech world. For Mark, this element has and continues to be “important for the growth of [the] language,” and has contributed to the vitality of these communities.

All in all the weekend was a great success, with especially memorable talks given by speakers working for Google, Trinket, and Hirease. Mark was impressed with the students’ enthusiasm and focus and lingered after both of iterations his talk to speak with attendees about their careers and interests. The next day he was equally affected by the range and talent behind Sunday’s hackathon projects as he followed the progress of various teams on Twitter. “These are the students [who] are going to help define what’s next,” he remarked.

Can’t get enough of Python, Django, and the talented Mark Lavin? Neither can we. Mark will be leading a workshop at PyCon on Building SMS Applications with Django along with fellow Cakti David Ray and our code school’s lead instructor, Caleb Smith. We’ll hope to see you there!

Django, pytz, NonExistentTimeError and AmbiguousTimeError

By Between engineering and real life from Django community aggregator: Community blog posts. Published on Mar 31, 2015.

Brief: In one of the project I work on we had to convert some old naive datetime objects to timezone aware ones. Converting naive datetime to timezone aware one is usually a straightforward job. In django you even have a nice utility function for this. For example:

import pytz from django.utils import timezone timezone.make_aware(datetime.datetime(2012, 3, 25, 3, 52), timezone=pytz.timezone('Europe/Stockholm')) # returns datetime.datetime(2012, 3, 25, 3, 52, tzinfo=<DstTzInfo 'Europe/Stockholm' CEST+2:00:00 DST>)

Problem: You can use this for quite a long time until one day you end up with something like this:

timezone.make_aware(datetime.datetime(2012, 3, 25, 2, 52), timezone=pytz.timezone('Europe/Stockholm')) # which leads to Traceback (most recent call last): File "", line 1, in File "/home/ilian/venvs/test/lib/python3.4/site-packages/django/utils/timezone.py", line 358, in make_aware return timezone.localize(value, is_dst=None) File "/home/ilian/venvs/test/lib/python3.4/site-packages/pytz/tzinfo.py", line 327, in localize raise NonExistentTimeError(dt) pytz.exceptions.NonExistentTimeError: 2012-03-25 02:52:00

Or this:

timezone.make_aware(datetime.datetime(2012, 10, 28, 2, 52), timezone=pytz.timezone('Europe/Stockholm')) #throws Traceback (most recent call last): File "", line 1, in File "/home/ilian/venvs/test/lib/python3.4/site-packages/django/utils/timezone.py", line 358, in make_aware return timezone.localize(value, is_dst=None) File "/home/ilian/venvs/test/lib/python3.4/site-packages/pytz/tzinfo.py", line 349, in localize raise AmbiguousTimeError(dt) pytz.exceptions.AmbiguousTimeError: 2012-10-28 02:52:00

Explanation: The reason for the first error is that in the real world this datetime does not exists. Due to the DST change on this date the clock jumps from 01:59 standard time to 03:00 DST. Fortunately (or not) pytz is aware of the fact that this time is invalid and will throw the exception above. The second exception is almost the same but it happens when switching from summer to standard time. From 01:59 DST the clock shifts to 01:00 standard time, so we end with a duplicate time.

Why has this happened(in our case)? Well we couldn't be sure how exactly this one got into our legacy data but the assumption is that at the moment when the record was saved the server has been in different timezone where this has been a valid time.

Solution 1: This fix is quite simple, just add an hour if the exception occurs.

try: date = make_aware( datetime.fromtimestamp(date_time, timezone=pytz.timezone('Europe/Stockholm')) ) except (pytz.NonExistentTimeError, pytz.AmbiguousTimeError): date = make_aware( datetime.fromtimestamp(date_time) + timedelta(hours=1), timezone=pytz.timezone('Europe/Stockholm') )

Solution 2: Instead of calling make_aware call timezone.localize directly.

try: date = make_aware( datetime.fromtimestamp(date_time, timezone=pytz.timezone('Europe/Stockholm')) ) except (pytz.NonExistentTimeError, pytz.AmbiguousTimeError): timezone = pytz.timezone('Europe/Stockholm') date = timezone.localize(datetime.fromtimestamp(date_time), is_dst=False)

The second solution probably needs some explanation. First lets check what make_aware does. The code bellow is take from Django's sourcecode as it is in version 1.7.7

def make_aware(value, timezone): """ Makes a naive datetime.datetime in a given time zone aware. """ if hasattr(timezone, 'localize'): # This method is available for pytz time zones. return timezone.localize(value, is_dst=None) else: # Check that we won't overwrite the timezone of an aware datetime. if is_aware(value): raise ValueError( "make_aware expects a naive datetime, got %s" % value) # This may be wrong around DST changes! return value.replace(tzinfo=timezone)

To simplify it, what Django does is to use the localize method of the timezone object(if it exists) to convert the datetime. When using pytz this localize method takes two arguments: the datetime value and is_dst. The last argument takes three possible values: None, False and True. When using None and the datetime matches the moment of the DST change pytz does not know how to handle the datetime and you get one of the exceptions shown above. False means that it should convert it to standard time and True that it should convert it to summer time.

Why isn't this fixed in Django? The simple answer is "because this is how it should work". For a bit longer check the respectful ticket.

Reminder: Do not forget that the "fix" above does not actually care whether the original datetime is during DST or not. In our case this was not criticla for our app, but in some other cases it might be, so use it carefully.

Thanks: Special thanks to Joshua who correctly pointed out in the comments that I have missed the AmbiguousTimeError in the original post which made me to look a bit more in the problem, research other solutions and update the article to its current content.

Django, pytz and NonExistentTimeError

By Between engineering and real life from Django community aggregator: Community blog posts. Published on Mar 30, 2015.

Brief: In one of the project I work on we have convert some old naive datetime object to timezone aware ones. Converting naive datetime to timezone aware one is usually straightforward job. In django you even have a nice utility function for this. For example:

import pytz from django.utils import timezone timezone.make_aware(datetime.datetime(2012, 3, 25, 3, 52), timezone=pytz.timezone('Europe/Stockholm')) # returns datetime.datetime(2012, 3, 25, 3, 52, tzinfo=<DstTzInfo 'Europe/Stockholm' CEST+2:00:00 DST>)

Problem: You can use this for quite a long time until one day you end up with something like this:

timezone.make_aware(datetime.datetime(2012, 3, 25, 2, 52), timezone=pytz.timezone('Europe/Stockholm')) # which leads to Traceback (most recent call last): File "", line 1, in File "/home/ilian/venvs/test/lib/python3.4/site-packages/django/utils/timezone.py", line 358, in make_aware return timezone.localize(value, is_dst=None) File "/home/ilian/venvs/test/lib/python3.4/site-packages/pytz/tzinfo.py", line 327, in localize raise NonExistentTimeError(dt) pytz.exceptions.NonExistentTimeError: 2012-03-25 02:52:00

Explanation: The reason for this error is that in the real world this datetime does not exists. Due to the DST change on this date the clock jumps from 01:59 directly to 03:00. Fortunately (or not) pytz is aware of the fact that this time is invalid and will throw the exception above.

Why this happens? Well we couldn't be sure how exactly this one got into our legacy data but the assumption is that at the moment when the record was saved the server has been in different timezone where this has been a valid time.

Solution: The fix is quite simple, just add an hour if the error occurs:

try: date = make_aware( datetime.fromtimestamp(date_time, timezone=pytz.timezone('Europe/Stockholm') ) except pytz.NonExistentTimeError: date = make_aware( datetime.fromtimestamp(date_time) + timedelta(hours=1), timezone=pytz.timezone('Europe/Stockholm') )

Tastypie with ForeignKey

By Agiliq Blog: Django web development from Django community aggregator: Community blog posts. Published on Mar 29, 2015.

Tastypie with ForeignKeys

This is a followup post on Getting started with tastypie. We will use the same project setup as used in the last post.

This post will cover:

  • Fetch ForeignKey data in GET calls
  • Create an object with ForeignKeys using POST calls

Setup the application

Let's add the capability to categorise the expenses

Add a model called ExpenseCategory

class ExpenseCategory(models.Model):
    name = models.CharField(max_length=100)
    description = models.TextField()

Add a FK from Expense to ExpenseCategory

class Expense(models.Model):
    description = models.CharField(max_length=100)
    amount = models.IntegerField()
    user = models.ForeignKey(User, null=True)
    category = models.ForeignKey(ExpenseCategory, null=True)

There already exists some Expense in db without an associated category, so make ExpenseCategory as nullable.

Create and apply migrations

python manage.py makemigrations
python manage.py migrate

Let's create an expensecategory from shell and associate it with an expense of user Sheryl.

u = User.objects.get(username='sheryl')
ec = ExpenseCategory.objects.create(name='Misc', description='Miscellaneous expenses')
e = Expense.objects.create(description='Went to Stockholm', amount='5000', user=u, category=ec)

Get FK fields in response too.

We want category in Expense GET endpoint too.

Our first approach would be adding 'category' to ExpenseCategory.Meta.fields. Try it

fields = ['description', 'amount', 'category']

Try the expense GET endpoint for Sheryl

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

Still don't see category in response. We need something more than this.

Adding fields.ForeignKey on ExpenseResource

There is no easy way to achieve this without adding a resource for ExpenseCategory.

We need to create an ExpenseCategoryResource similar to ExpenseResource

Add ExpenseCategoryResource to expenses/api.py

class ExpenseCategoryResource(ModelResource):
    class Meta:
        queryset = ExpenseCategory.objects.all()
        resource_name = 'expensecategory'

Add proper url pattern for ExpenseCategoryResource in expenses/api.py

expense_category_resource = ExpenseCategoryResource()

urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^api/', include(expense_resource.urls)),
    url(r'^api/', include(expense_category_resource.urls)),
)

Verify things are properly setup for ExpenseCategoryResource by accessing

http://localhost:8000/api/expensecategory/?format=json

Add the following to ExpenseCategory

category = fields.ForeignKey(ExpenseCategoryResource, attribute='category', null=True)

Try

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

After this you'll be able to see category in response

This will return resource_uri of ExpenseCategory by default

Using full=True

Probably you want to see the name and description of category in the response

Make the following modification

category = fields.ForeignKey(ExpenseCategoryResource, attribute='category', null=True, full=True)

Try the GET endpoint again

http://localhost:8000/api/expense/?username=sheryl&api_key=1a23&format=json

POST data with FK

There are several ways in which we can set category on expense while making POST call to create expenses.

Post with resource_uri of FK

We already have one ExpenseCategory in the db and the resource_uri for that expensecategory is '/api/expensecategory/1/'

We want to create an expense and set the category as our earlier created expensecategory.

post_data = {'description': 'Bought a phone for testing', 'amount': 2200, 'category': '/api/expensecategory/1/'}
post_url = 'http://localhost:8000/api/expense/?username=sheryl&api_key=1a23'
r = requests.post(post_url, data=json.dumps(post_data), headers=headers)

Posting entire data of FK

You find that the expense you want to create doesn't fit in any of the existing categories. You want to create a new expensecategory while making POST data to expense endpoint.

So we want to creating ExpenseCategory and Expense together.

You need to post the following data in such case.

post_data = {'description': 'Went to paris to attend a conference', 'amount': 9000, 'category': {'name': 'Travel', 'description': 'Expenses incurred on travelling'}}

No category exists for Travel yet.

Check the count of ExpenseCategory currently so that later you can verify that a new ExpenseCategory is created.

ExpenseCategory.objects.count()
1   #output

POST the data to expense endpoint

r = requests.post(post_url, data=json.dumps(post_data), headers=headers)
print r.status_code    #401
Why you got 401

Even though you tried creating an Expense on expense post endpoint, tastypie internally tries creating an expensecategory because of structure of post_data. But tastypie finds that ExpenseCategoryResource doesn't have authorization to allow POST yet.

So we need to add proper authorization to ExpenseCategory before this post call can succeed.

Add the following to ExpenseCategoryResource.Meta

authorization = Authorization()

POSTing again

Try the post call again.

r = requests.post(post_url, data=json.dumps(post_data), headers=headers)

This would have worked well and a new ExpenseCategory should have been created.

ExpenseCategory.objects.count()
2    #output

Also the new expense would have got associated with the newly created ExpenseCategory.

Welcome to Our New Staff Members

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Mar 27, 2015.

We’ve hit one of our greatest growth points yet in 2015, adding nine new team members since January to handle our increasing project load. There are many exciting things on the horizon for Caktus and our clients, so it’s wonderful to have a few more hands on deck.

One of the best things about working at Caktus is the diversity of our staff’s interests and backgrounds. In order of their appearance from left to right in the photos above, here’s a quick look at our new Cakti’s roles and some fun facts:

Neil Ashton

Neil was also a Caktus contractor who has made the move to full-time Django developer. He is a keen student of more than programming languages; he holds two degrees in Classics and another Master’s in Linguistics.

Jeff Bradberry

Though Jeff has been working as a contractor at Caktus, he recently became a full-time developer. In his spare time, he likes to play around with artificial intelligence, sometimes giving his creations a dose of inexplicable, random behavior to better mimic us poor humans.

Ross Pike

Ross is our new Lead Designer and has earned recognition for his work from Print, How Magazine, and the AIGA. He also served in the Peace Corps for a year in Bolivia on a health and water mission.

Lucas Rowe

Lucas joins us for six months as a game designer, courtesy of a federal grant to reduce the spread of HIV. When he’s not working on Epic Allies, our HIV medication app, he can be found playing board games or visiting local breweries.

Erin Mullaney

Erin has more than a decade of development experience behind her, making her the perfect addition to our team of Django developers. She loves cooking healthy, vegan meals and watching television shows laden with 90s nostalgia.

Liza Chabot

Liza is an English major who loves to read, write, and organize, all necessary skills as Caktus’ Administrative and Marketing Assistant. She is also a weaver and sells and exhibits her handwoven wall hangings and textiles in the local craft community.

NC Nwoko

NC’s skills are vast in scope. She graduated from UNC Chapel Hill with a BA in Journalism and Mass Communication with a focus on public relations and business as well as a second major in International Studies with a focus on global economics. She now puts this experience to good use as Caktus’ Digital Health Product Manager, but on the weekends you can find her playing video games and reading comic books.

Edward Rowe

Edward is joining us for six months as a game developer for the Epic Allies project. He loves developing games for social good. Outside of work, Edward continues to express his passion for games as an avid indie game developer, UNC basketball fan, and board and video game player.

Rob Lineberger

Rob is our new Django contractor. Rob is a renaissance man; he’s not only a skilled and respected visual artist, he’s trained in bioinformatics, psychology, information systems and knows his way around the kitchen.

To learn more about our team, visit our About Page. And if you’re wishing you could spend your days with these smart, passionate people, keep in mind that we’re still hiring.

Welcome to Our New Staff Members

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Mar 27, 2015.

We’ve hit one of our greatest growth points yet in 2015, adding nine new team members since January to handle our increasing project load. There are many exciting things on the horizon for Caktus and our clients, so it’s wonderful to have a few more hands on deck.

One of the best things about working at Caktus is the diversity of our staff’s interests and backgrounds. In order of their appearance from left to right in the photos above, here’s a quick look at our new Cakti’s roles and some fun facts:

Neil Ashton

Neil was also a Caktus contractor who has made the move to full-time Django developer. He is a keen student of more than programming languages; he holds two degrees in Classics and another Master’s in Linguistics.

Jeff Bradberry

Though Jeff has been working as a contractor at Caktus, he recently became a full-time developer. In his spare time, he likes to play around with artificial intelligence, sometimes giving his creations a dose of inexplicable, random behavior to better mimic us poor humans.

Ross Pike

Ross is our new Lead Designer and has earned recognition for his work from Print, How Magazine, and the AIGA. He also served in the Peace Corps for a year in Bolivia on a health and water mission.

Lucas Rowe

Lucas joins us for six months as a game designer, courtesy of a federal grant to reduce the spread of HIV. When he’s not working on Epic Allies, our HIV medication app, he can be found playing board games or visiting local breweries.

Erin Mullaney

Erin has more than a decade of development experience behind her, making her the perfect addition to our team of Django developers. She loves cooking healthy, vegan meals and watching television shows laden with 90s nostalgia.

Liza Chabot

Liza is an English major who loves to read, write, and organize, all necessary skills as Caktus’ Administrative and Marketing Assistant. She is also a weaver and sells and exhibits her handwoven wall hangings and textiles in the local craft community.

NC Nwoko

NC’s skills are vast in scope. She graduated from UNC Chapel Hill with a BA in Journalism and Mass Communication with a focus on public relations and business as well as a second major in International Studies with a focus on global economics. She now puts this experience to good use as Caktus’ Digital Health Product Manager, but on the weekends you can find her playing video games and reading comic books.

Edward Rowe

Edward is joining us for six months as a game developer for the Epic Allies project. He loves developing games for social good. Outside of work, Edward continues to express his passion for games as an avid indie game developer, UNC basketball fan, and board and video game player.

Rob Lineberger

Rob is our new Django contractor. Rob is a renaissance man; he’s not only a skilled and respected visual artist, he’s trained in bioinformatics, psychology, information systems and knows his way around the kitchen.

To learn more about our team, visit our About Page. And if you’re wishing you could spend your days with these smart, passionate people, keep in mind that we’re still hiring.

High Performance Django Infrastructure Preview

By Lincoln Loop from Django community aggregator: Community blog posts. Published on Mar 26, 2015.

One of the most common requests we've heard since releasing our book, High Performance Django is: "Do you have more code/configuration examples?" It's a pretty loaded question because the book covers everything from Python code to deploying and configuring servers. After some thinking on how to deliver this in a format people could easily understand, I realized the answer was already right under our noses.

We've been users of Salt for configuration management for almost three years. Over the last few weeks I've been extracting our internal Salt states into a reusable and extensible system I like to call "infrastructure-in-a-box". It encompasses all the lessons we've learned over the years with our different clients and allows anyone to setup a complete Python website (load balancer, web accelerator, cache, database, task queue, etc.) in about 15 minutes. The exact same code can be used to setup a single Vagrant machine on your laptop or a ten server farm in the cloud. I whipped together a quick screencast preview of it in action (apologies for the low-quality audio):

I'm really excited about being able to offer this as a companion product to our book. It's going to save people a lot of time and money (not to mention heartache) figuring it out on their own.

Here's the thing though, I need your feedback to get this released. I know this would have been useful for us and for many of our clients, but there's a lot of work left to take it from where it is to a polished product. Is this something that interests you? What topics would you like to see included? Leave a comment or send us an email with your thoughts. Thanks!

RevSys Roundup - March 2015

By Revolution Systems Blog from Django community aggregator: Community blog posts. Published on Mar 26, 2015.

RevSys Roundup - March 2015

How to automatically and professionally remove photo backgrounds

By Cloudinary Blog - Django from Django community aggregator: Community blog posts. Published on Mar 26, 2015.

RTB add-on It is common for e-commerce, media, and news sites to remove image backgrounds or make them transparent in order to place the main element of the image on either white or color backgrounds. The final result better integrates an image into a site or specific page’s graphic design. For example, a fashion site that presents clothes or shoes should have the main element of a photo (e.g. shoes) extracted from the original image background, then edited to fit the site’s catalogue design and structure.

Remove-The-Background add-on

RTB logo

We are glad to introduce the Remove-The-Background editing add-on, a third party image processing add-on that supports image background removal. This add-on is brought to you by Remove-The-Background, a leading vendor of image editing solution components, including professional background removal, that is performed by a team of human experts. We, at Cloudinary, have tried it multiple times and the results were pretty impressive.

There are automatic tools that can aid in background removal. Nonetheless, if your goal is to create perfect results, utilizing a graphic editor/designer would be your best bet. However, instead of hiring an in-house or freelance designer, Cloudinary’s Remove-The-Background add-on makes this process much simpler. Since the new add-on is fully integrated into Cloudinary's image management pipeline), when you upload an image, you can easily and automatically have it edited by Remove-The-Background experts.

How to remove a photo background with Cloudinary

We’d like to demonstrate this process, starting with the picture below:

Ruby:
cl_image_tag("shoes_orig.jpg")
PHP:
cl_image_tag("shoes_orig.jpg")
Python:
CloudinaryImage("shoes_orig.jpg").image()
Node.js:
cloudinary.image("shoes_orig.jpg")
Java:
cloudinary.url().imageTag("shoes_orig.jpg")
jQuery:
$.cloudinary.image("shoes_orig.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("shoes_orig.jpg")
Original shows image

You can begin the process either while the photo is being uploaded to Cloudinary, using the upload API demonstrated in the code sample below, or by using the Admin API for images that have already been uploaded. Simply specify the background_removal parameter in either API.

Ruby:
Cloudinary::Uploader.upload("shoes.jpg",
  :public_id => "shoes",
  :background_removal => 'remove_the_background')
PHP:
\Cloudinary\Uploader::upload("shoes.jpg", 
  array(
    "public_id" => "shoes",
    "background_removal" => "remove_the_background"
  ));
Python:
cloudinary.uploader.upload("shoes.jpg",
  public_id = "shoes",
  background_removal = "remove_the_background")
Node.js:
cloudinary.uploader.upload("shoes.jpg", 
  function(result) { console.log(result); }, 
  { public_id: "shoes",
    background_removal: "remove_the_background" });
Java:
cloudinary.uploader().upload("shoes.jpg", Cloudinary.asMap(
  "public_id", "shoes",
  "background_removal", "remove_the_background"));

As mentioned above, the actual background removal is performed by Remove-The-Background’s team of experts and it could therefore take up to 24 hours to complete. Cloudinary processes the request asynchronously, then when the background removal is complete, the original uploaded image is replaced by an edited one. A backup of the original image is automatically saved to Cloudinary. It is also possible to receive a notification that indicates when the editing process is complete. Below, you can see how the picture's background was removed with great results:

Ruby:
cl_image_tag("shoes.jpg")
PHP:
cl_image_tag("shoes.jpg")
Python:
CloudinaryImage("shoes.jpg").image()
Node.js:
cloudinary.image("shoes.jpg")
Java:
cloudinary.url().imageTag("shoes.jpg")
jQuery:
$.cloudinary.image("shoes.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("shoes.jpg")
Resulting image with background removed

Pictures can be further manipulated to fit your own graphics and design using Cloudinary's manipulation URLs. For example, below, you can see the same image cropped to 250 x 250, with increased saturation.

Ruby:
cl_image_tag("shoes.jpg", :width=>250, :height=>250, :crop=>:fill, :effect=>"saturation:80")
PHP:
cl_image_tag("shoes.jpg", array("width"=>250, "height"=>250, "crop"=>"fill", "effect"=>"saturation:80"))
Python:
CloudinaryImage("shoes.jpg").image(width=250, height=250, crop="fill", effect="saturation:80")
Node.js:
cloudinary.image("shoes.jpg", {width: 250, height: 250, crop: "fill", effect: "saturation:80"})
Java:
cloudinary.url().transformation(new Transformation().width(250).height(250).crop("fill").effect("saturation:80")).imageTag("shoes.jpg")
jQuery:
$.cloudinary.image("shoes.jpg", {width: 250, height: 250, crop: "fill", effect: "saturation:80"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(250).Height(250).Crop("fill").Effect("saturation:80")).BuildImageTag("shoes.jpg")
250x250 cropped shows image with background removed

This add-on can remove the background from any type of photo, including pictures of people.

Ruby:
cl_image_tag("woman.jpg")
PHP:
cl_image_tag("woman.jpg")
Python:
CloudinaryImage("woman.jpg").image()
Node.js:
cloudinary.image("woman.jpg")
Java:
cloudinary.url().imageTag("woman.jpg")
jQuery:
$.cloudinary.image("woman.jpg")
.Net:
cloudinary.Api.UrlImgUp.BuildImageTag("woman.jpg")
Original woman photo Woman photo with background removed

The images below have been dynamically created using Cloudinary's manipulation URLs. 200x200 face-detection based thumbnails were created. The image on the left is a thumbnail of the original image while the image on the right is a thumbnail with the background removed.

Ruby:
cl_image_tag("woman.jpg", :width=>200, :height=>200, :crop=>:thumb, :gravity=>:face)
PHP:
cl_image_tag("woman.jpg", array("width"=>200, "height"=>200, "crop"=>"thumb", "gravity"=>"face"))
Python:
CloudinaryImage("woman.jpg").image(width=200, height=200, crop="thumb", gravity="face")
Node.js:
cloudinary.image("woman.jpg", {width: 200, height: 200, crop: "thumb", gravity: "face"})
Java:
cloudinary.url().transformation(new Transformation().width(200).height(200).crop("thumb").gravity("face")).imageTag("woman.jpg")
jQuery:
$.cloudinary.image("woman.jpg", {width: 200, height: 200, crop: "thumb", gravity: "face"})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation().Width(200).Height(200).Crop("thumb").Gravity("face")).BuildImageTag("woman.jpg")
Thumbnail of original woman photo Thumbnail of woman photo with background removed

Remove The Background supports additional editing profiles that can be specified via Cloudinary’s API (e.g. keep/remove shadow, transparent/white background, and more). Please contact us if you need a custom editing profile. For more details about this add-on check out our Remove-The-Background add-on documentation.

Final Notes

Cloudinary’s Remove-The-Background add-on helps preserve your site or app’s professional look, without the need for in-house graphic designers or long and complex editing processes. Customers of the Basic plan or higher can try the Remove-The-Background add-on for free then later subscribe to a plan that best meets your specific requirements.

If you don't have a Cloudinary account yet, sign up for a free account here.

Twitter way back machine

By Thierry Schellenbach from Django community aggregator: Community blog posts. Published on Mar 24, 2015.

One of the associates at Techstars created this beautiful demo of Stream’s technology (github repo). If you ever wondered about Thomas Edison’s or Nikola Tesla’s tweets, check it out! :)

Share and Enjoy: Digg Sphinn del.icio.us Facebook Mixx Google

Getting started with Django tastypie

By Agiliq Blog: Django web development from Django community aggregator: Community blog posts. Published on Mar 23, 2015.

Django tastypie is a library to write RESTful apis in Django.

Why use REST

You have a database backed web application. This application tracks expenses. The application allows the capability to enter your expenses, view all your expenses, delete an expense etc. Essentially this application provides CRUD functionality. Django application has access to database credentials, but they are never seen by the users of the web application. Django application decides what to show to which user. Django application ensures that a particular user only sees the expenses entered by him and not somebody else's expenses.

Now you want to provide a mobile application (Android or iOS) corresponding to this web application. Android application should allow the user to view his expenses, create an expense as well as any other CRUD functionality. But database credentials could not be put in Android code as it is not too hard to decompile an apk and get the db credentials. And we never want a user to get the db credentials else he will be in a position to see everyone's expenses and the entire database. So there has to be another way to allow mobile applications to get things from the database. This is where REST comes into picture.

With REST, we have three components. A database, a Django application and a mobile application. Mobile application never accesses the database directly. It makes a REST api call to Django application. Mobile application also sends a api_key specific to the mobile user. Based on api_key, Django application determines what data to make visible to this particular api_key owner and sends the corresponding data in response.

Resource

REST stands for Representational State Transfer. It is a standard for transferring the state of a Resource, from web to mobile.

What do I mean by state of a Resource?

An expense could be a resource. A Person could be a resource. A blog post could be a resource. Basically any object or instance your program deals with could be a resource. And a resource's state is maintained in it's attributes. eg: You could have a model called Expense. The state of a expense instance is represented by its attributes.

Any REST library should be able to create and return a representation of such resource, which simply stated means that REST library should be able to tell us the attributes and their values for different model instances. And tastypie is adept at doing this.

Setting up the application

I am using Django 1.7. Some things might be different for you if you are using different version of Django.

As with all projects, I want to keep things in a virtual environment

$ mkvirtualenv tastier
$ workon tastier

Install Django

$ pip install Django

Start a Django project

(tastier) $ django-admin.py startproject tastier

(tastier) $ cd tastier/

Start an app

(tastier) $ python manage.py startapp expenses

Add this app to INSTALLED_APPS

Run migration

(tastier)~ $ python manage.py migrate

Runserver

(tastier)~ $ python manage.py runserver

Check that your are able to access http://localhost:8000/admin/login/

I have pushed the code for this project to Github. You will be able to checkout at different commits in the project to see specific things.

Getting started

Install django-tastypie.

(tastier) $ pip install django-tastypie

Create a file called expenses/api.py where you will keep all the tastypie related things.

Suppose your program deals with a resource called Expense. Let's create a model Expense in expenses/models.py

class Expense(models.Model):
    description = models.CharField(max_length=100)
    amount = models.IntegerField()

Run migrations

python manage.py makemigrations
python manage.py migrate

We will later add a ForeignKey(User) to Expense to associate an expense with User. Don't worry about it for now, we will come back to it.

Let's add few Expense instances in the database.

Expense.objects.create(description='Ate pizza', amount=100)
Expense.objects.create(description='Went to Cinema', amount=200)

Handling GET

You want the ability to get the representation of all expenses in your program at url "http://localhost:8000/api/expenses/".

To deal with a resource, tastypie requires a class which overrides ModelResource. Let's call our class ExpenseResource. Add following to expenses/api.py

from tastypie.resources import ModelResource

from .models import Expense

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expense'

And you need to add the following to tastier/urls.py

from expenses.api import ExpenseResource

expense_resource = ExpenseResource()

urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^api/', include(expense_resource.urls)),
)

GET all expenses

After this you should be able to hit

http://localhost:8000/api/expense/?format=json

and you will see all the expenses from database in the response.

The response would be:

{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 2}, "objects": [{"amount": 100, "description": "Ate pizza", "id": 1, "resource_uri": "/api/expense/1/"}, {"amount": 200, "description": "Went to Cinema", "id": 2, "resource_uri": "/api/expense/2/"}]}

You will find the representation of expense instances in objects key of response.

Get a particular expense

You can get the representation of expense with id 1 at

http://localhost:8000/api/expense/1/?format=json

See how you are able to hit these two urls without adding them in urlpatterns. These urlpatterns are added by tastypie internally.

How these endpoints help and ties with mobile application example?

If the mobile app wants to show all the expenses it could use the url http://localhost:8000/api/expense/?format=json, get the response, parse the response and show the result on app.

Right now every user will see all the expenses. As we move forward we will see how only a user's expenses will be returned when a REST call is made from his/her mobile device.

Serialization

You must have realized that REST returns you serialized data. You might be wondering why use django-tastypie to achieve it, and not just use json.dumps. You can undoubtedly use json.dumps and not use django-tastypie to provide REST endpoints. But django-tastypie allows the ability to do many more things very easily as you will soo agree. Just hang on.

Changing Meta.resource_name

You can change ExpenseResource.Meta.resource_name from expense to expenditure.

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expenditure'

And then the old urls will stop working. Your new GET urls in that case will be

http://localhost:8000/api/expenditure/?format=json
http://localhost:8000/api/expenditure/1/?format=json

Changing the resource_name changes the urls tastypie makes available to you.

Now change the resource_name back to expense.

We have our first commit at this point. You can checkout to this commit to see the code till this point.

git checkout b6a9c6

Meta.fields

Suppose you only want description in expense representation, but don't want amount. So you can add a fields attribute on ExpenseResource.Meta

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expenditure'
    fields = ['description']

Try

http://localhost:8000/api/expense/?format=json

So if you don't have fields attribute on Meta, all the attributes of Model will be sent in response. If you have fields, only attributes listed in fields will be sent in response.

Let's add amount also to fields. Though this gives us the same behaviour as not having ExpenseResource.Meta.fields at all.

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expenditure'
    fields = ['description', 'amount']

We have our second commit at this point. You can checkout till this point by doing:

git checkout 61194c

Filtering

Suppose you only want the Expenses where amount exceeds 150.

If we had to do this with Django model we would say:

Expense.objects.filter(amount__gt=150)

amount__gt is the key thing here. This could be appended to our url pattern to get the expenses where amount exceeds 150.

This could be achieved at url

http://localhost:8000/api/expense/?amount__gt=150&format=json

Try this. You will get an error because we haven't asked tastypie to allow filtering yet.

Add filtering attribute to ExpenseResource.Meta

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expense'
    fields = ['description', 'amount']
    filtering = {
        'amount': ['gt']
    }

You should be able to use

http://localhost:8000/api/expense/?amount__gt=150&format=json

This will only return the expenses where amount exceeds 150.

Now we want to get all the expenses on Pizza. We could get pizza expenses in following way from shell.

Expense.objects.filter(description__icontains='pizza')

So to achieve this thing in api, we need to make following changes to ExpenseResource.Meta.filtering:

class Meta:
    queryset = Expense.objects.all()
    resource_name = 'expense'
    fields = ['description', 'amount']
    filtering = {
        'amount': ['gt'],
        'description': ['icontains']
    }

And then following url would give us the pizza expenses

http://localhost:8000/api/expense/?description__icontains=pizza&format=json

With GET endpoints we were able to do the Read operations. With POST we will be able to do Create operations, as we will see in next section.

Handling POST

It's hard to do POST from the browser. So we will use requests library to achieve this.

Check expense count before doing POST.

>>> Expense.objects.count()
2

Tastypie by default doesn't authorize a person to do POST request. The default authorization class is ReadOnlyAuthorization which allows GET calls but doesn't allow POST calls. So you will have to disallow authorization checks for the time being. Add the following to ExpenseResource.Meta

authorization = Authorization()

You'll need to import Authorization class for it.

from tastypie.authorization import Authorization

After this, ExpenseResource would look like:

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expense'
        fields = ['description', 'amount']
        filtering = {
            'amount': ['gt'],
            'description': ['icontains']
        }
        authorization = Authorization()

Don't get into detail of Authorization for now, I will come back to it.

Let's make a POST request to our rest endpoint which will create an Expense object in the database.

post_url = 'http://localhost:8000/api/expense/'
post_data = {'description': 'Bought first Disworld book', 'amount': 399}
headers = {'Content-type': 'application/json'}
import requests
import json
r = requests.post(post_url, json.dumps(post_data), headers=headers)
>>> print r.status_code
201

status_code 201 means that your Expense object was properly created. You can also verify it by checking that Expense count increased by 1.

>>> Expense.objects.count()
3

If you hit the GET endpoint from your browser, you will see this new Expense object too in the response. Try

http://localhost:8000/api/expense/?format=json

We have third commit at this point.

git checkout 749cf3

Explanation of POST

  • You need to POST at the same url where you get all the expenses. Compare the two urls.
  • One way of posting is to POST json encoded data. So we used json.dumps
  • If you are sending json encoded data, you need to send appopriate Content-type header too.


How this ties in with mobile

Android or iOS has a way to make POST request at a given url with headers. So you tell mobile app about the endpoint where they need to post and the data to post. They will call this rest endpoint, and the posted data will be handled by Django tastypie and proper row will be created in the database.

Adding authentication

Currently GET and POST endpoints respond to every request. So even users who aren't registered with the site will be able to see the expenses. Our first step is ensuring that only registered users are able to use the GET endpoints.

Api tokens and sessions

In a web application, a user logs in once and then she is able to make any number of web requests without being asked to login every time. eg: User logs in once and then can see her expense list page. After first request she can refresh the page, and can still get response without being asked for her login credentials again. This works because Django uses sessions and cookies to store user state. So browser sends a cookie to Django everytime the user makes a request, and Django app can associate the cookie with a user and shows the data for this particular user.

With mobile apps, there is no concept of sessions, unless the mobile is working with a WebView(clarify with Shabda). The session corresponding thing in a mobile app is Api key. So an api key is associated with a user. Every REST call should include this api key, and then tastypie can use this key to verify whether a logged in user is making the request.

Creating user and api token

Let's create an user in our system and a corresponding api token for her.

On a shell

u = User.objects.create_user(username='sheryl', password='abc', email='sheryl@abc.com')

Tastypie provides a model called ApiKey which allows storing tokens for users. Let's create a token for Sheryl.

from tastypie.models import ApiKey
ApiKey.objects.create(key='1a23', user=u)

We are setting the api token for sheryl as '1a23'

You need to ensure tastypie is in INSTALLED_APPS and you have migrated before you could create ApiKey instance.

The default authentication class provided by tastypie is Authentication which allows anyone to make GET requests. We need to set ExpenseResource.Meta.authentication to ensure that only users who provide valid api key are able to get response from GET endpoints.

Add the following on ExpensesResource.Meta.

authentication = ApiKeyAuthentication()

You need to import ApiKeyAuthentication.

from tastypie.authentication import ApiKeyAuthentication

Try the GET endpoint to get the list of expenses

http://localhost:8000/api/expense/?format=json

You will not see anything in response. If you see your runserver terminal, you'll notice that status code 401 is raised.

Api key should be sent in the request to get proper response.

Try the following url

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

With this Sheryl will be able to get proper api response.

Try sending wrong api_key for sheryl and you will not see proper response.

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a2

With these we ensure that only registered users of the system with proper api key will be able to make GET requests.

Fourth commit at this point

git checkout 48725f

How this ties in with mobile app

When user installs the app, he logs in using his username and password for first time. These credentials are sent to Django server using a REST call. Django server returns the api key corresponding to this user to the mobile app. Mobile app stores this api token on mobile end and then uses this token for every subsequent REST call. User doesn't have to provide the credentials anymore.

Making unauthenticated POST requests

Unauthenticated POST requests will not work anymore

Try creating an Expense without passing any api key.

post_data = {'description': 'Bought Two scoops of Django', 'amount': 399}
headers = {'Content-type': 'application/json'}
r = requests.post("http://localhost:8000/api/expense/", data=json.dumps(post_data), headers=headers)
print r.status_code       #This will give 401

Check that Expense count isn't increased

Making authenticated POST requests

You only need to change the url to include username and api_key in the url. This will make the request authenticated.

r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(post_data), headers=headers)

This should have worked and Expense count should have increased.

Try with wrong api_key and it will fail.

Getting only User's expense

Till now we aren't associating Expense to User. Let's add a ForeignKey to User from Expense.

Expense model becomes:

from django.db import models
from django.contrib.auth.models import User


class Expense(models.Model):
    description = models.CharField(max_length=100)
    amount = models.IntegerField()
    user = models.ForeignKey(User, null=True)

Since we already have some Expenses in db which aren't associated with a User, so we kept User as a nullable field.

Make and run migrations

python manage.py makemigrations
python manage.py migrate

Right now our authorization class is set to Authorization. With this every user is authorized to see every expense. We will have to add a custom authorization class to enforce that users see only their expenses.

Add the following to expenses/api.py

class ExpenseAuthorization(Authorization):

    def read_list(self, object_list, bundle):
        return object_list.filter(user=bundle.request.user)

And change authorization on ExpenseResource.Meta so it becomes:

class ExpenseResource(ModelResource):

    class Meta:
        queryset = Expense.objects.all()
        resource_name = 'expense'
        fields = ['description', 'amount']
        filtering = {
            'amount': ['gt'],
            'description': ['icontains']
        }
        authorization = ExpenseAuthorization()
        authentication = ApiKeyAuthentication()

Explanation of ExpenseAuthorization

  • When GET endpoint is called for expense list, object_list is created which gives all the expenses.
  • After this, authorization is checked where further filtering could be done.
  • In case of GET on list endpoint, authorization class' read_list() method is called. object_list is passed to read_list.
  • In tastypie there is a variable called bundle. And bundle has access to request using bundle.request
  • When authentication is used properly, bundle.request.user is populated with correct user.

Try expense list endpoint for Sheryl

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

You will not get any expense after adding ExpenseAuthorization

{"meta": {"limit": 20, "next": null, "offset": 0, "previous": null, "total_count": 0}, "objects": []}

This happenned because at this point no expense is associated with Sheryl.

Create an expense for Sheryl and try the GET endpoint

On the shell

u = User.objects.get(username='sheryl')
Expense.objects.create(description='Paid for the servers', amount=1000, user=u)
Expense.objects.create(description='Paid for CI server', amount=500, user=u)

Try expense list endpoint for Sheryl again

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

You should be able to see all of Sheryl's expenses.

Fifth commit here.

git checkout 26f7c1

How mobile app will use it.

When Sheryl installs the app, she will be asked to login for the first time. There will be a REST endpoint which takes the username and password for a user and if the credentials are right, returns the api key for the user. Sheryl's api key will be returned to the mobile app which will store it in local storage. And when Sheryl wants to see her expenses, this REST call will be made to Django server.

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

This will only return Sheryl's expenses.

POST and create Sheryl's expense

Till now if a POST request is made, even if with Sheryl's api key, expense is created in db but is not associated with Sheryl.

We want to add functionality where if POST request is made from Sheryl's device then expense is associated with Sheryl. If POST request is made from Mark's device then expense should be associated with Mark.

Tastypie provides several hookpoints. We will use one such hookpoint. ModelResource provides a method called hydrate which we need to override. Add the following method to ExpenseResource.

def hydrate(self, bundle):
    bundle.obj.user = bundle.request.user
    return bundle
  • This method is called during POST/PUT calls.
  • bundle.obj is an Expense instance about to be saved in the database.
  • So we set user on bundle.obj by reading it from bundle.request. We have already discussed how bundle.request.obj is populated during authentication flow.

Make a POST request now with Sheryl's api_key.

post_data = {'description': 'Paid for iDoneThis', 'amount': 700}
r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(post_data), headers=headers)

Verify that the latest expense instance gets associated with Sheryl. You can also verify it by seeing that this object gets returned in GET expense list endpoint.

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

Sixth commit at this point

git checkout 17b932

Try on your own

  • Create one more user in database, from shell.
  • Create api key for this user.
  • POST to REST endpoint with this new user's api_key and username and verify that the expense gets associated with this new user.
  • Check GET expense list for this new user and verify that only expense created for this user is in the response.

Now is a good time to dig deeper into django-tastypie and understand about following:

  • Dehydrate cycle. It is used during GET calls.
  • Hydrate cycle. It is used during POST/PUT calls. Once you read about hydrate cycle, you will understand when method hydrate() is called.
  • More about authorization and different methods available on Authorization which could be overridden by you.

Want more?

I am still trying few things with tastypie. Hereafter I will not have much explanation, but I will point to the commit where I attain certain functionality change.

Authorization on detail endpoint.

Expense with id 1 is not associated with any user. But Sheryl is still able to see it at:

http://localhost:8000/api/expense/1/?format=json&username=sheryl&api_key=1a23

She shouldn't be able to see it as it is not her expense.

So add the following to ExpenseAuthorization

def read_detail(self, object_list, bundle):
    obj = object_list[0]
    return obj.user == bundle.request.user

After this Sheryl will not be able to see detail endpoint of any expense which doesn't belong to her. Try it

http://localhost:8000/api/expense/1/?format=json&username=sheryl&api_key=1a23

Commit id for this:

e650f3
git show e650f3

PUT endpoint

Expense with id 5 belongs to Sheryl. She wants to update this expense, she essentially want to change the description.

Current thing is:

http://localhost:8000/api/expense/5/?format=json&username=sheryl&api_key=1a23

Make PUT request

put_url = "http://localhost:8000/api/expense/5/?username=sheryl&api_key=1a23"
put_data = {'description': 'Paid for Travis'}
headers = {'Content-type': 'application/json'}
r = requests.put(put_url, data=json.dumps(put_data), headers=headers)

Description of Expense 5 is updated as you can verify by trying the detail endpoint again.

http://localhost:8000/api/expense/5/?format=json&username=sheryl&api_key=1a23

Notice that amount remains unchanged. So PUT changes whatever data you provide in the api call and lets everything else remain as it is.

DELETE endpoint

First check all of Sheryl's expenses

http://localhost:8000/api/expense/?format=json&username=sheryl&api_key=1a23

Sheryl wants to delete her expense with id 5. After this is done she will have one less expense in db.

delete_url = "http://localhost:8000/api/expense/5/?username=sheryl&api_key=1a23"
r = requests.delete(delete_url)

Verify that this expense got deleted.

So we were able to do Create, Read, Update and Delete with REST api calls.

Restrict POST request to certain users only

Suppose we want the users to be able to create expenses from web end but don't want to allow creating expense from mobile using the api. Yeah, weird requiremtn.

Also we don't want to disallow POST for all users. We still want Sheryl to be able to POST.

To try this we first need a new user with api key in our system. Create it from Django shell.

u = User.objects.create_user(username='mark', password='def', email='mark@abc.com')
ApiKey.objects.create(key='2b34', user=u)

Restricting POST for everyone except Sheryl

Add following method to ExpenseAuthorization

def create_detail(self, object_list, bundle):
    user = bundle.request.user
    # Return True if current user is Sheryl else return False
    return user.username == "sheryl"

Try making POST request as Mark and see you will not be able to do it. If you want you can see the expense count at this point.

post_data = {'description': 'Petty expense', 'amount': 3000}
r = requests.post("http://localhost:8000/api/expense/?username=mark&api_key=2b34", data=json.dumps(post_data), headers=headers)
print r.status_code #Should have got 401

Also you can check the expense count again to verify that expense isn't created.

Status code 401 tells that you aren't authorized to do this operation.

Verify that Sheryl is still able to create expense

Try posting the same post_data as Sheryl

r = requests.post("http://localhost:8000/api/expense/?username=sheryl&api_key=1a23", data=json.dumps(post_data), headers=headers)
print r.status_code

Status code must be 201 in this case which means expense is created. You will be able to see this expense at Sheryl's GET expense list endpoint.

Verify that Mark is still able to do GET

Mark or any other user should still be able to make GET request even if he isn't able to make POST request.

http://localhost:8000/api/expense/?username=mark&api_key=2b34&format=json

Since Mark doesn't have any expense in db, so no object is there is objects key of response. Try creating an expense for this user from shell and then try the GET endpoint again.

Astro Code School Now Accepting Applications - Intermediate Django + Python

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Mar 20, 2015.

 

Code

 

I'm really happy to officially announce the first Python and Django Web Engineering class at Astro Code School. I’ll outline some details here and you can also find them on our classes page.

This class is twelve weeks long and full time Monday to Friday from 9 AM – 5 PM. It'll be taught here at the Astro Code School at 108 Morris Street, Suite 1b, Durham, NC. We will conduct two Python and Django Web Engineering classes in 2015. The first one in term two starts May 18, 2015 and ends August 10, 2015. The second one in term three starts September 22, 2015 and ends December 15, 2015.

Enrollment for both sections opens today March 20. There is space for twelve students in each class. More information about the enrollment process is on our Apply page. Part of that process is an entrance exam that is designed to ensure you're ready to succeed. The price per person for Python and Django Web Engineering is $12,000.

The Python and Django Web Engineering class is intended for intermediate level students. Its goal is to help you start your career as a backend web engineer. To start down this path we recommend you prepare yourself. A few things you can do are: read some books on Python & Django, complete the Django Girls tutorial, watch videos on Youtube, and take an online class or two in Python.

 

 

Python and Django make a powerful team to build maintainable web applications quickly. When you take this course you will build your own web application during lab time with assistance from your teacher and professional Django developers. You’ll also receive help preparing your portfolio and resume to find a job using the skills you’ve learned.

Here's the syllabus:

  1. Python Basics, Git & GitHub, Unit Testing
  2. Object Oriented Programming, Functional Programming, Development Process, Command Line
  3. HTML, HTTP, CSS, LESS, JavaScript, DOM
  4. Portfolio Development, Intro to Django, Routing, Views, Templates
  5. SQL, Models, Migrations, Forms, Debugging
  6. Django Admin, Integrating Apps, Upgrading Django, Advanced Django
  7. Ajax, JavaScript, REST
  8. Linux Admin, AWS, Django Deployment, Fabric
  9. Interviewing Skills, Computer Science Topics, Review
  10. Final Project Labs
  11. Final Project Labs
  12. Final Project Labs

 

This comprehensive course is taught by experienced developer and trained teacher Caleb Smith. He's been working full time at Caktus Consulting Group, the founders of Astro Code School and the nation’s largest Django firm. He’s worked on many client projects over the years. He’s also applied his experience as a former public school teacher to teach Girl Develop It Python classes and as an adjunct lecturer at the University of North Carolina-Chapel Hill. I think you'll really enjoy working with and learning from Caleb. He's a wonderful person.

For the past six months we've been working very hard to launch the school. A large amount of our time has been spent on a application to receive our license from the State of North Carolina to conduct a proprietary school. As of today Astro is one of two code schools in North Carolina that have received this license. We found it a very important task to undertake. It helped us do our due diligence to run a honest and fair school that will protect the rights of students who will be attending Astro Code School. This long process also explains why we've waited to tell you all the details. We're required to wait till we have a license to open our application process.

Thanks for checking out Astro Code School. If you have any questions please contact me.

Astro Code School Now Accepting Applications - Intermediate Django + Python

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Mar 20, 2015.

 

Code

 

I'm really happy to officially announce the first Python and Django Web Engineering class at Astro Code School. I’ll outline some details here and you can also find them on our classes page.

This class is twelve weeks long and full time Monday to Friday from 9 AM – 5 PM. It'll be taught here at the Astro Code School at 108 Morris Street, Suite 1b, Durham, NC. We will conduct two Python and Django Web Engineering classes in 2015. The first one in term two starts May 18, 2015 and ends August 10, 2015. The second one in term three starts September 22, 2015 and ends December 15, 2015.

Enrollment for both sections opens today March 20. There is space for twelve students in each class. More information about the enrollment process is on our Apply page. Part of that process is an entrance exam that is designed to ensure you're ready to succeed. The price per person for Python and Django Web Engineering is $12,000.

The Python and Django Web Engineering class is intended for intermediate level students. Its goal is to help you start your career as a backend web engineer. To start down this path we recommend you prepare yourself. A few things you can do are: read some books on Python & Django, complete the Django Girls tutorial, watch videos on Youtube, and take an online class or two in Python.

 

 

Python and Django make a powerful team to build maintainable web applications quickly. When you take this course you will build your own web application during lab time with assistance from your teacher and professional Django developers. You’ll also receive help preparing your portfolio and resume to find a job using the skills you’ve learned.

Here's the syllabus:

  1. Python Basics, Git & GitHub, Unit Testing
  2. Object Oriented Programming, Functional Programming, Development Process, Command Line
  3. HTML, HTTP, CSS, LESS, JavaScript, DOM
  4. Portfolio Development, Intro to Django, Routing, Views, Templates
  5. SQL, Models, Migrations, Forms, Debugging
  6. Django Admin, Integrating Apps, Upgrading Django, Advanced Django
  7. Ajax, JavaScript, REST
  8. Linux Admin, AWS, Django Deployment, Fabric
  9. Interviewing Skills, Computer Science Topics, Review
  10. Final Project Labs
  11. Final Project Labs
  12. Final Project Labs

 

This comprehensive course is taught by experienced developer and trained teacher Caleb Smith. He's been working full time at Caktus Consulting Group, the founders of Astro Code School and the nation’s largest Django firm. He’s worked on many client projects over the years. He’s also applied his experience as a former public school teacher to teach Girl Develop It Python classes and as an adjunct lecturer at the University of North Carolina-Chapel Hill. I think you'll really enjoy working with and learning from Caleb. He's a wonderful person.

For the past six months we've been working very hard to launch the school. A large amount of our time has been spent on a application to receive our license from the State of North Carolina to conduct a proprietary school. As of today Astro is one of two code schools in North Carolina that have received this license. We found it a very important task to undertake. It helped us do our due diligence to run a honest and fair school that will protect the rights of students who will be attending Astro Code School. This long process also explains why we've waited to tell you all the details. We're required to wait till we have a license to open our application process.

Thanks for checking out Astro Code School. If you have any questions please contact me.

Test django view with cookies

By Andrey Zhukov's blog from Django community aggregator: Community blog posts. Published on Mar 20, 2015.

To test some view which use cookies:

1
2
3
4
5
6
7
8
9
10
from Cookie import SimpleCookie
from django import test

class SomeTest(test.TestCase):

  def test_some_view(self):
      self.client.cookies = SimpleCookie({'test_cookie': 'test_value'})
      response = self.client.get('/some-url/')

      self.assertEqual(response.client.cookies['test_cookie'].value, 'test_value')

Transform your image overlays with on-the-fly manipulation

By Cloudinary Blog - Django from Django community aggregator: Community blog posts. Published on Mar 19, 2015.

Layer apply Front end developers may want to combine multiple images into a single image. For example, when creating and adding watermarks to stock photos, adding shapes or badges, preparing content for print (e.g. placing a logo on a t-shirt or a mug), adding a caption, and so on.

Multiple images can be combined by overlaying them one on top of the other. However, since it is not a given that both the underlying and overlaying images match each other, let alone your graphic design, you may need to perform further manipulations (e.g. resize, crop, change colors, create a better fit). This is where Cloudinary comes in.

Cloudinary's image overlay feature helps users easily combine multiple images. It supports image and text overlays using on-the-fly manipulation URLs. In this blog post, we will show you how to separately manipulate, process, and transform underlying and overlaying images, then dynamically generate a resulting image that you can embed on your site.

Manipulating image overlays

Suppose you have a website that sells personalized gifts. Users can upload their own photos, add text, and your site will automatically crop and manipulate those photos and text on the gift of their choice. For example, a couple may want to place their picture on a coffee mug. This would require you to resize and manipulate both the underlying image of the coffee mug and the overlaying picture of the couple until they fit together in harmony. Once the images are put in place, you can add text and perform further manipulations if necessary.

Below is an example of the original images of the couple and coffee mug that were uploaded to the cloud for further manipulation and delivery.

Coffee cup Nice couple

You can add an image overlay using Cloudinary's overlay parameter (or l for URLs). Returning to our example, here is what the final version of the coffee mug would look like with the overlaying picture of the couple:

Ruby:
cl_image_tag("coffee_cup.jpg", :transformation=>[
  {:width=>400, :height=>250, :crop=>:fill, :gravity=>:south},
  {:overlay=>"nice_couple", :width=>90, :x=>-20, :y=>18, :gravity=>:center}
  ])
PHP:
cl_image_tag("coffee_cup.jpg", array("transformation"=>array(
  array("width"=>400, "height"=>250, "crop"=>"fill", "gravity"=>"south"),
  array("overlay"=>"nice_couple", "width"=>90, "x"=>-20, "y"=>18, "gravity"=>"center")
  )))
Python:
CloudinaryImage("coffee_cup.jpg").image(transformation=[
  {"width": 400, "height": 250, "crop": "fill", "gravity": "south"},
  {"overlay": "nice_couple", "width": 90, "x": -20, "y": 18, "gravity": "center"}
  ])
Node.js:
cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 90, x: -20, y: 18, gravity: "center"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(400).height(250).crop("fill").gravity("south").chain()
  .overlay("nice_couple").width(90).x(-20).y(18).gravity("center")).imageTag("coffee_cup.jpg")
jQuery:
$.cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 90, x: -20, y: 18, gravity: "center"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(400).Height(250).Crop("fill").Gravity("south").Chain()
  .Overlay("nice_couple").Width(90).X(-20).Y(18).Gravity("center")).BuildImageTag("coffee_cup.jpg")
Image overlay

Transformation instructions are used to perform the image manipulations using dynamic URLs. Cloudinary's client libraries assist in building these URLs. You can apply multiple chained transformations by separating them with a slash / in your image's URL.

In order to better manipulate image overlays, you can set the flags parameter to layer_apply (or fl_layer_apply for URLs), which then tells Cloudinary that all chained transformations that were specified up until the flag, are to be applied on the overlaying image instead of the containing image.

Using our coffee mug example, below you can see how we have applied multiple manipulations both on the containing image as well as the overlay. The containing image has been cropped to fill a 400x250 rectangle and the overlaying image of the couple has been cropped using face detection. Color saturation has been increased by 50% and the vignette effect has been applied. Finally, the resulting image has been resized to 100 pixels wide, converted to a circular shape and positioned with 20 pixels offset from the center of the containing image.

Ruby:
cl_image_tag("coffee_cup.jpg", :transformation=>[
  {:width=>400, :height=>250, :crop=>:fill, :gravity=>:south},
  {:overlay=>"nice_couple", :width=>1.3, :height=>1.3, :crop=>:crop, :gravity=>:faces, :flags=>:region_relative},
  {:effect=>"saturation:50"},
  {:effect=>"vignette"},
  {:radius=>"max", :width=>100, :x=>-20, :y=>20, :gravity=>:center, :flags=>:layer_apply}
  ])
PHP:
cl_image_tag("coffee_cup.jpg", array("transformation"=>array(
  array("width"=>400, "height"=>250, "crop"=>"fill", "gravity"=>"south"),
  array("overlay"=>"nice_couple", "width"=>1.3, "height"=>1.3, "crop"=>"crop", "gravity"=>"faces", "flags"=>"region_relative"),
  array("effect"=>"saturation:50"),
  array("effect"=>"vignette"),
  array("radius"=>"max", "width"=>100, "x"=>-20, "y"=>20, "gravity"=>"center", "flags"=>"layer_apply")
  )))
Python:
CloudinaryImage("coffee_cup.jpg").image(transformation=[
  {"width": 400, "height": 250, "crop": "fill", "gravity": "south"},
  {"overlay": "nice_couple", "width": 1.3, "height": 1.3, "crop": "crop", "gravity": "faces", "flags": "region_relative"},
  {"effect": "saturation:50"},
  {"effect": "vignette"},
  {"radius": "max", "width": 100, "x": -20, "y": 20, "gravity": "center", "flags": "layer_apply"}
  ])
Node.js:
cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 1.3, height: 1.3, crop: "crop", gravity: "faces", flags: "region_relative"},
  {effect: "saturation:50"},
  {effect: "vignette"},
  {radius: "max", width: 100, x: -20, y: 20, gravity: "center", flags: "layer_apply"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(400).height(250).crop("fill").gravity("south").chain()
  .overlay("nice_couple").width(1.3).height(1.3).crop("crop").gravity("faces").flags("region_relative").chain()
  .effect("saturation:50").chain()
  .effect("vignette").chain()
  .radius("max").width(100).x(-20).y(20).gravity("center").flags("layer_apply")).imageTag("coffee_cup.jpg")
jQuery:
$.cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 1.3, height: 1.3, crop: "crop", gravity: "faces", flags: "region_relative"},
  {effect: "saturation:50"},
  {effect: "vignette"},
  {radius: "max", width: 100, x: -20, y: 20, gravity: "center", flags: "layer_apply"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(400).Height(250).Crop("fill").Gravity("south").Chain()
  .Overlay("nice_couple").Width(1.3).Height(1.3).Crop("crop").Gravity("faces").Flags("region_relative").Chain()
  .Effect("saturation:50").Chain()
  .Effect("vignette").Chain()
  .Radius("max").Width(100).X(-20).Y(20).Gravity("center").Flags("layer_apply")).BuildImageTag("coffee_cup.jpg")
Image overlay with further manipulation

Learn more about Cloudinary’s image manipulation capabilities

Manipulating multiple image overlays

In addition to being able to manipulate a single image overlay, Cloudinary allows you to add and manipulate multiple overlays, as well. You can do this by chaining another overlay, setting the flags parameter to layer_apply (or fl_layer_apply for URLs), and applying multiple image transformations. Adding another overlay is as simple as manipulating a picture to suit the existing underlying and overlaying images. In our coffee mug example, we added a balloon as an additional overlay and performed the following manipulations: resized to be 30 pixels wide, changed the hue level to pink, and rotated it five degrees.

Ruby:
cl_image_tag("coffee_cup.jpg", :transformation=>[
  {:width=>400, :height=>250, :crop=>:fill, :gravity=>:south},
  {:overlay=>"nice_couple", :width=>1.3, :height=>1.3, :crop=>:crop, :gravity=>:faces, :flags=>:region_relative},
  {:effect=>"saturation:50"},
  {:effect=>"vignette"},
  {:radius=>"max", :width=>100, :x=>-20, :y=>20, :gravity=>:center, :flags=>:layer_apply},
  {:overlay=>"balloon", :width=>30},
  {:angle=>5, :effect=>"hue:-20"},
  {:x=>30, :y=>5, :flags=>:layer_apply}
  ])
PHP:
cl_image_tag("coffee_cup.jpg", array("transformation"=>array(
  array("width"=>400, "height"=>250, "crop"=>"fill", "gravity"=>"south"),
  array("overlay"=>"nice_couple", "width"=>1.3, "height"=>1.3, "crop"=>"crop", "gravity"=>"faces", "flags"=>"region_relative"),
  array("effect"=>"saturation:50"),
  array("effect"=>"vignette"),
  array("radius"=>"max", "width"=>100, "x"=>-20, "y"=>20, "gravity"=>"center", "flags"=>"layer_apply"),
  array("overlay"=>"balloon", "width"=>30),
  array("angle"=>5, "effect"=>"hue:-20"),
  array("x"=>30, "y"=>5, "flags"=>"layer_apply")
  )))
Python:
CloudinaryImage("coffee_cup.jpg").image(transformation=[
  {"width": 400, "height": 250, "crop": "fill", "gravity": "south"},
  {"overlay": "nice_couple", "width": 1.3, "height": 1.3, "crop": "crop", "gravity": "faces", "flags": "region_relative"},
  {"effect": "saturation:50"},
  {"effect": "vignette"},
  {"radius": "max", "width": 100, "x": -20, "y": 20, "gravity": "center", "flags": "layer_apply"},
  {"overlay": "balloon", "width": 30},
  {"angle": 5, "effect": "hue:-20"},
  {"x": 30, "y": 5, "flags": "layer_apply"}
  ])
Node.js:
cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 1.3, height: 1.3, crop: "crop", gravity: "faces", flags: "region_relative"},
  {effect: "saturation:50"},
  {effect: "vignette"},
  {radius: "max", width: 100, x: -20, y: 20, gravity: "center", flags: "layer_apply"},
  {overlay: "balloon", width: 30},
  {angle: 5, effect: "hue:-20"},
  {x: 30, y: 5, flags: "layer_apply"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(400).height(250).crop("fill").gravity("south").chain()
  .overlay("nice_couple").width(1.3).height(1.3).crop("crop").gravity("faces").flags("region_relative").chain()
  .effect("saturation:50").chain()
  .effect("vignette").chain()
  .radius("max").width(100).x(-20).y(20).gravity("center").flags("layer_apply").chain()
  .overlay("balloon").width(30).chain()
  .angle(5).effect("hue:-20").chain()
  .x(30).y(5).flags("layer_apply")).imageTag("coffee_cup.jpg")
jQuery:
$.cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 1.3, height: 1.3, crop: "crop", gravity: "faces", flags: "region_relative"},
  {effect: "saturation:50"},
  {effect: "vignette"},
  {radius: "max", width: 100, x: -20, y: 20, gravity: "center", flags: "layer_apply"},
  {overlay: "balloon", width: 30},
  {angle: 5, effect: "hue:-20"},
  {x: 30, y: 5, flags: "layer_apply"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(400).Height(250).Crop("fill").Gravity("south").Chain()
  .Overlay("nice_couple").Width(1.3).Height(1.3).Crop("crop").Gravity("faces").Flags("region_relative").Chain()
  .Effect("saturation:50").Chain()
  .Effect("vignette").Chain()
  .Radius("max").Width(100).X(-20).Y(20).Gravity("center").Flags("layer_apply").Chain()
  .Overlay("balloon").Width(30).Chain()
  .Angle(5).Effect("hue:-20").Chain()
  .X(30).Y(5).Flags("layer_apply")).BuildImageTag("coffee_cup.jpg")
Multiple image overlays with manipulations

Manipulating text overlays

Cloudinary supports adding dynamic text overlays in any style of customized text. What's more, the text overlay can be further manipulated, just like image overlays. So, returning to our example, we have now added a text overlay that we have colored, using the colorize effect, and rotated.

Ruby:
cl_image_tag("coffee_cup.jpg", :transformation=>[
  {:width=>400, :height=>250, :crop=>:fill, :gravity=>:south},
  {:overlay=>"nice_couple", :width=>1.3, :height=>1.3, :crop=>:crop, :gravity=>:faces, :flags=>:region_relative},
  {:effect=>"saturation:50"},
  {:effect=>"vignette"},
  {:radius=>"max", :width=>100, :x=>-20, :y=>20, :gravity=>:center, :flags=>:layer_apply},
  {:overlay=>"balloon", :width=>30},
  {:angle=>5, :effect=>"hue:-20"},
  {:x=>30, :y=>5, :flags=>:layer_apply},
  {:color=>"#f08", :overlay=>"text:Cookie_40_bold:Love", :effect=>"colorize"},
  {:angle=>20, :x=>-45, :y=>44, :flags=>:layer_apply}
  ])
PHP:
cl_image_tag("coffee_cup.jpg", array("transformation"=>array(
  array("width"=>400, "height"=>250, "crop"=>"fill", "gravity"=>"south"),
  array("overlay"=>"nice_couple", "width"=>1.3, "height"=>1.3, "crop"=>"crop", "gravity"=>"faces", "flags"=>"region_relative"),
  array("effect"=>"saturation:50"),
  array("effect"=>"vignette"),
  array("radius"=>"max", "width"=>100, "x"=>-20, "y"=>20, "gravity"=>"center", "flags"=>"layer_apply"),
  array("overlay"=>"balloon", "width"=>30),
  array("angle"=>5, "effect"=>"hue:-20"),
  array("x"=>30, "y"=>5, "flags"=>"layer_apply"),
  array("color"=>"#f08", "overlay"=>"text:Cookie_40_bold:Love", "effect"=>"colorize"),
  array("angle"=>20, "x"=>-45, "y"=>44, "flags"=>"layer_apply")
  )))
Python:
CloudinaryImage("coffee_cup.jpg").image(transformation=[
  {"width": 400, "height": 250, "crop": "fill", "gravity": "south"},
  {"overlay": "nice_couple", "width": 1.3, "height": 1.3, "crop": "crop", "gravity": "faces", "flags": "region_relative"},
  {"effect": "saturation:50"},
  {"effect": "vignette"},
  {"radius": "max", "width": 100, "x": -20, "y": 20, "gravity": "center", "flags": "layer_apply"},
  {"overlay": "balloon", "width": 30},
  {"angle": 5, "effect": "hue:-20"},
  {"x": 30, "y": 5, "flags": "layer_apply"},
  {"color": "#f08", "overlay": "text:Cookie_40_bold:Love", "effect": "colorize"},
  {"angle": 20, "x": -45, "y": 44, "flags": "layer_apply"}
  ])
Node.js:
cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 1.3, height: 1.3, crop: "crop", gravity: "faces", flags: "region_relative"},
  {effect: "saturation:50"},
  {effect: "vignette"},
  {radius: "max", width: 100, x: -20, y: 20, gravity: "center", flags: "layer_apply"},
  {overlay: "balloon", width: 30},
  {angle: 5, effect: "hue:-20"},
  {x: 30, y: 5, flags: "layer_apply"},
  {color: "#f08", overlay: "text:Cookie_40_bold:Love", effect: "colorize"},
  {angle: 20, x: -45, y: 44, flags: "layer_apply"}
  ]})
Java:
cloudinary.url().transformation(new Transformation()
  .width(400).height(250).crop("fill").gravity("south").chain()
  .overlay("nice_couple").width(1.3).height(1.3).crop("crop").gravity("faces").flags("region_relative").chain()
  .effect("saturation:50").chain()
  .effect("vignette").chain()
  .radius("max").width(100).x(-20).y(20).gravity("center").flags("layer_apply").chain()
  .overlay("balloon").width(30).chain()
  .angle(5).effect("hue:-20").chain()
  .x(30).y(5).flags("layer_apply").chain()
  .color("#f08").overlay("text:Cookie_40_bold:Love").effect("colorize").chain()
  .angle(20).x(-45).y(44).flags("layer_apply")).imageTag("coffee_cup.jpg")
jQuery:
$.cloudinary.image("coffee_cup.jpg", {transformation: [
  {width: 400, height: 250, crop: "fill", gravity: "south"},
  {overlay: "nice_couple", width: 1.3, height: 1.3, crop: "crop", gravity: "faces", flags: "region_relative"},
  {effect: "saturation:50"},
  {effect: "vignette"},
  {radius: "max", width: 100, x: -20, y: 20, gravity: "center", flags: "layer_apply"},
  {overlay: "balloon", width: 30},
  {angle: 5, effect: "hue:-20"},
  {x: 30, y: 5, flags: "layer_apply"},
  {color: "#f08", overlay: "text:Cookie_40_bold:Love", effect: "colorize"},
  {angle: 20, x: -45, y: 44, flags: "layer_apply"}
  ]})
.Net:
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(400).Height(250).Crop("fill").Gravity("south").Chain()
  .Overlay("nice_couple").Width(1.3).Height(1.3).Crop("crop").Gravity("faces").Flags("region_relative").Chain()
  .Effect("saturation:50").Chain()
  .Effect("vignette").Chain()
  .Radius("max").Width(100).X(-20).Y(20).Gravity("center").Flags("layer_apply").Chain()
  .Overlay("balloon").Width(30).Chain()
  .Angle(5).Effect("hue:-20").Chain()
  .X(30).Y(5).Flags("layer_apply").Chain()
  .Color("#f08").Overlay("text:Cookie_40_bold:Love").Effect("colorize").Chain()
  .Angle(20).X(-45).Y(44).Flags("layer_apply")).BuildImageTag("coffee_cup.jpg")
Multiple image and text overlays with further manipulation

Summary

Cloudinary’s powerful capabilities allow you to manipulate and generate complex, combined images that match your graphic design requirements. You can use Cloudinary's dynamic manipulation URLs with user-uploaded images in order to manipulate the images while combining multiple images and text overlays into a single new image. With the new features introduced in this post, you can apply Cloudinary’s rich set of manipulation capabilities separately on each layer of underlying or overlaying images or text. All overlay manipulation features are available with all Cloudinary plans, including the free tier. If you don't have a Cloudinary account yet, sign up for a free account here.

Why RapidSMS for SMS Application Development

By Caktus Consulting Group from Django community aggregator: Community blog posts. Published on Mar 16, 2015.

Caktus has been involved in quite a few projects (Libyan voter registration, UNICEF Project Mwana, and several others) that include text messaging (a.k.a. Short Message Service, or SMS), and we always use RapidSMS as one of our tools. We've also invested our own resources in supporting and extending RapidSMS.

There are other options; why do we consistently choose RapidSMS?

What is RapidSMS

First, what is RapidSMS? It's an open source package of useful tools that extend the Django web development framework to support processing text messages. It includes:

  • A framework for writing code to be invoked when a text message is received and respond to it
  • A set of backends - pluggable code modules that can interface to various ways of connecting your Django program to the phone network to pass text messages back and forth
  • Sample applications
  • Documentation

The backends are required because unlike email, there's no universal standard for sending and receiving text messages over the Internet. Often we get access to the messages via a third party vendor, like Twilio or Tropo, that provides a proprietary interface. RapidSMS isolates us from the differences among vendors.

RapidSMS is open source, under the BSD license, with UNICEF acting as holder of the contributors' agreements (granting a license for RapidSMS to use and distribute their contributions). See the RapidSMS license for more about this.

Alternatives

Here are some of the alternatives we might have chosen:

  • Writing from scratch: starting each project new and building the infrastructure to handle text messages again
  • Writing to a particular vendor's API: writing code that sends and receives text messages using the programming interface provided by one of the online vendors that provide that service, then building applications around that
  • Other frameworks

Why RapidSMS

Why did we choose RapidSMS?

  • RapidSMS builds on Django, our favorite web development framework.
  • RapidSMS is at the right level for us. It provides components that we can use to build our own applications the way we need to, and the flexibility to customize its behavior.
  • RapidSMS is open source, under the BSD license. There are no issues with our use of it, and we are free to extend it when we need to for a particular project. We then have the opportunity to contribute our changes back to the RapidSMS community.
  • RapidSMS is vendor-neutral. We can build our applications without being tied to any particular vendor of text messaging services. That's good for multiple reasons:
  • We don't have to pick a vendor before we can start.
  • We could change vendors in the future without having to rewrite the applications.
  • We can deploy applications to different countries that might not have any common vendor for messaging services.

It's worth noting that using RapidSMS doesn't even require using an Internet text messaging vendor. We can use other open source applications like Vumi or Kannel as a gateway to provide us with even more options:

  • use hardware called a "cellular/GSM modem" (basically a cell phone with a connection to a computer instead of a screen)
  • interface directly to a phone company's own servers over the Internet, using several widely used protocols

Summary

RapidSMS is a good fit for us at Caktus, it adds a lot to our projects, and we've been pleased to be able to contribute back to it.

Caktus will be leading a workshop on building RapidSMS applications during PyCon 2015 on Tuesday, April 7th 3:00-5:30.

Is Open Source Consulting Dead?

By chrism from . Published on Sep 10, 2013.

Has Elvis left the building? Will we be able to sustain ourselves as open source consultants?

Consulting and Patent Indemification

By chrism from . Published on Aug 09, 2013.

Article about consulting and patent indemnification

Python Advent Calendar 2012 Topic

By chrism from . Published on Dec 24, 2012.

An entry for the 2012 Japanese advent calendar at http://connpass.com/event/1439/

Why I Like ZODB

By chrism from . Published on May 15, 2012.

Why I like ZODB better than other persistence systems for writing real-world web applications.

A str. __iter__ Gotcha in Cross-Compatible Py2/Py3 Code

By chrism from . Published on Mar 03, 2012.

A bug caused by a minor incompatibility can remain latent for long periods of time in a cross-compatible Python 2 / Python 3 codebase.

In Praise of Complaining

By chrism from . Published on Jan 01, 2012.

In praise of complaining, even when the complaints are absurd.

2012 Python Meme

By chrism from . Published on Dec 24, 2011.

My "Python meme" replies.

In Defense of Zope Libraries

By chrism from . Published on Dec 19, 2011.

A much too long defense of Pyramid's use of Zope libraries.

Plone Conference 2011 Pyramid Sprint

By chrism from . Published on Nov 10, 2011.

An update about the happenings at the recent 2011 Plone Conference Pyramid sprint.

Jobs-Ification of Software Development

By chrism from . Published on Oct 17, 2011.

Try not to Jobs-ify the task of software development.

WebOb Now on Python 3

By chrism from . Published on Oct 15, 2011.

Report about porting to Python 3.

Open Source Project Maintainer Sarcastic Response Cheat Sheet

By chrism from . Published on Jun 12, 2011.

Need a sarcastic response to a support interaction as an open source project maintainer? Look no further!

Pylons Miniconference #0 Wrapup

By chrism from . Published on May 04, 2011.

Last week, I visited the lovely Bay Area to attend the 0th Pylons Miniconference in San Francisco.

Pylons Project Meetup / Minicon

By chrism from . Published on Apr 14, 2011.

In the SF Bay Area on the 28th, 29th, and 30th of this month (April), 3 separate Pylons Project events.

PyCon 2011 Report

By chrism from . Published on Mar 19, 2011.

My personal PyCon 2011 Report