Skip to content. | Skip to navigation

Personal tools
Log in
Sections
You are here: Home

Latest Plone Posts

Surprise features that you didn't ask for - Mike Haworth

From Planet Plone. Published on Sep 13, 2014.

Server Side Request Forgery

  • Could access services on localhost
  • Access other hosts in DMZ
  • Bypass host-based authentication systems

Vulnerable systems:

  • memcached
  • couchdb

Urllib: accepts file://

Pycurl accepts gopher://

Memcached: gopher://127.0.0.1:11211/2get

Preventing: blacklist is error prone.

XML External Entity (XXE):

Safer: http://pypi.python.org/pypi/diffusedxml

Signed cookies - session data is visible to user.

Also unless it’s got discount applied, just reapply discount. Can have a unique number that is issued each time we give discount.

Don’t use pickle as could execute shellcode but instead use JSON.

Pentesting challenges: http://tinyurl.com/h4ckit

Automated Deployment with Ansible - Juergen Brendel

From Planet Plone. Published on Sep 13, 2014.

Configuration - system packages, settings, users, apps

CM Tools - describe the desired state. E.g. ensure all system packages are updated.

Puppet, chef - feel bit like working with Java.

Salt, Ansible - simple.

Ansible - orchestration engine. Written in Python. Uses YAML to write “playbooks”. Can use it to configure Windows.

No central configuration server/client (on target).

Just needs SSH access and Python on target machine.

Modules - 100s of them. They know how to do stuff.

Deletes module code from target after running it on target.

Can define groups to operate against.

group_vars directory/ all, groupname1

Ansible ‘cloud’ modules: AWS, Docker.

Modules: all output to stdout + JSON.

https://github.com/jbrendel/ansible_cluster_setup

Lightning Talks

From Planet Plone. Published on Sep 13, 2014.

LCAuckland

LCA: 12-16 January 2015

Program online in next year or 2.

Singapore also put in proposal for future LCA.

6-700 attendees hoped.

Heritage Preserve #WeWantJam

Grabbed 30TBs of .co.nz internet.

2 days of catering, hacking.

If you’d like to do it again tweet #WeWantJam

Less Boilerplate

import argparse

Command line

http://github.com/aliles/begins

Command line programs for busy developers.

Show us the world

rbenv - isolate different

pyenv shell 3.4.1

pyenv virtualenv name

pyenv local dirname

MySQL performance schema

Debugging schema for the database.

Can configure at runtime.

5.5 or 6 dbs.

Julia

Very pythonic. Stolen lots of good ideas from Python.

Compiled just in time. Timing a file that does nothing it takes 2.25s.

The Python Promotion Pamphlet

International Python Promotion Pamphlet. Glossy & stylised.

Creating NZ-version. Will have high-quality printable version and web-version.

Make your company more visible to Python programmers rather than pay recruitment agents.

Blackbox

http://sheltered-forest-9460.herokuapp.com

http://github.com/tuxbert/blackbox

NZPUG

When you subscribe to the mailing list (607 members on mailing list) you aren’t a member of the incorporated society.

Jessica McKellar’s Kiwi PyCon73,000 views. Far more than Guido McV’s keynote at PyCon US.

Thomi is commandeering LT and announcing my leaving committee and asking if someone can join NZPUG committee. See committee members if would like to join.

Docker for Python

Will do to IT industry what container box did to shipping industry.

LXC (Linux Lightweight Containers) + UnionFS.

Docker file is essentially like a bash script file of commands.

You can inherit from other docker files.

Fig - Vagrant for Docker.

Ship deltas not images.

Shipyard - 3rd party ecosystem.

Gotchas

  • Use docker-osx, PyCharm

Object Factories

Sole object is to create objects for testing purposes.

Don’t care about what object is. Just assign random attributes to objects.

Code Obfuscation

Weird bugs: object() > object()

If doing tests against time it will fail at midnight depending on your timezone.

False = True

Arduino + Thermal Printer + Python

Arduino sketch, Python script, Git

http://github.com/nick-NZ/Arduino-pull-request-printer

Dynamically creating Python tutorials and presentations

Massey Computer Science moved to Python in 2011.

Emacs: an operating system disguised as an editor.

http://www.orgmode.org

Org-babel - generate code.

Org-slidy

Bad-ass Postgres Tricks

http://bit.ly/pgtrix1

  1. Template Databases - clone dbs
  2. Estimated Counts
  3. Hstore - values have to be strings
  4. Smart Indexes
  5. Schema Change Locks
  6. PL/Python

"One button" test and deploy on AWS

Use Vagrant+Ansible

Scripted creation of AMI and deployment.

Python Antipatterns

Better: list comprehension

Better: raise AssertionError

Better: if pattern in input_str

Don’t pass empty list as default are.

if 5 <= i < 10: print “something”

Better: return NotImplementedError

Map reduction using Python scripts

Script to create ArcMap maps.

Making weird maps with Python

Pulled data from census + meshblock boundaries (Koordinates.com) + map box (tile mill) + GDAL.

Should have used Fiona + Shapely.

Multimedia programming using Gstreamer (and, of course, Python) - Douglas Bagnall

From Planet Plone. Published on Sep 13, 2014.

Pipelines consist of linked elements.

Elements:

  • Link
  • Source
  • Sink (file, net, screen, etc)
  • Demux, tee

$ gst-launch-1.0 filesrc location=video.ogv ! oggdemux ! theoradec ! autovideosink

playbin - just play file = mplayer

Probably about 100 media players in world that are based on gstreamer

Muxer: combining 2 streams into 1 file.

Really good example - video wall.

Decode bin -> split into 4 streams (tee) -> queues (often freezes as all running in 1 thread and bump into eachother) -> crop (I.e. Remove 3/4s of video). And also play audio.

Can just use gstreamer cli syntax when calling gi calls in Python.

github: douglasbagnall

http://gstreamer.freedesktop.org

Understanding human language with Python - Alyona Medelyan

From Planet Plone. Published on Sep 13, 2014.

Android Auto: hands-free operation via voice commands.

WordLense: “augmented reality translation”

"Her" movie in reality: "Siri will you marry me? Siri doesn’t seem to be available."

Segmentation complexities.

Disambiguation complexities. E.g. Flying planes can be dangerous.

Categories - taxonomy terms, entities - biological/chemical, sentiment.

NKTK - Python toolkit for NLP.

How to get to the core words? Remove stopwords.

from nltk.corpus import stopwords from nltk.tokenizie import word_tokenize

Keyword scoring: TFxIDF

Gensim:

from gensim import corpora, models

When ranking words can use score to discard them.

Text Categorisation with NLTK

Sentiment Analysis:

from textblob import TextBlob

http://deeplearning.net/software/theano

http://textblob.readthedocs.org

http://scikit-learn.org/stable

http://nltk.org

http://radimrehurek.com/gensim

http://github.com/zelandiya/KiwiPyCon-NLP-tutorial

PyPy.js: What? How? Why? - Ryan Kelly

From Planet Plone. Published on Sep 13, 2014.

@rfkelly

Mozilla - protecting & promoting the web.

The Web? Technology + Culture (open - don’t need to submit to ‘web pool’, ubiquitous, secure, trustworthy)

"For Mozilla, anything that the Web can’t do, or anything that the Web is not faster and better at than native technologies, is a bug. We should file it in our Bugzilla system, so we can start writing a patch to fix it" - Andreas Gal (Mozilla CEO)

var vm = new PyPyJS() import js js.eval()

An experiment in building a fast, compliant, in-browser Python environment.

PyPy.js is … Not so Fast. Incompatibilities between the JIT compilers.

Is … Humongous.

How? PyPy + Emscripten.

Emscripten: an LLVM backend that generates JavaScript.

"It’s awful all the way down"

Alts: http://GitHub.com/PythonJS

"Please, file bugs against the Web"

http://pypyjs.org

http://github.com/rfk/talk-pypyjs-what-how-why

Deploying a Django application using Juju - Tim Penhey

From Planet Plone. Published on Sep 13, 2014.

Or "How I used Juju to deploy my Django app up to AWS"

Used backbone.js

Juju is written in Go. Service orchestration in the cloud.

Charms:

  • Encapsulate application configurations
  • Define how services get deployed

Existing Charms

  • PostgreSQL
  • Python-Django
  • Gunicorn

Juju can help with scaling later.

Simple repeatable deployments.

Writing a subordinate charm.

Hooks are just scripts. Used Install hooks and Everything else hooks.

Promoted charms are only ones that are officially available.

"As a Juju core developer it was interesting that if I didn’t know the value that Juju would give me later, I would’ve thrown it out ages ago."

Insufficient documentation around using the Python-Django charm, had to read the code.

Insufficient documentation around how to write a payload charm for Python-Django, very much work it out as you go.

Python-Django and gunicorn still needs nginx to be really useful.

Should have a full stack Django charm.

juju bootstrap|deploy|add-relation|deploy|expose

2nd Keynote: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy - Francesco Biscani

From Planet Plone. Published on Sep 13, 2014.

Max Planck Institute for Astronomy, European Space Agency.

PaGMO and Astropy - will mainly talk about PaGMO as wrote this and more familiar with this.

PaGMO - Parallel Global Multiobjective Optimiser. Optimisation tool. Initially a trajectory optimisation tool, evolved as a general-purpose optimiser.

Focused on parallel and distributed computing. Can use via C++ or Python.

Astropy - community effort to develop a single core package for Astronomy.

Optimisation - a large area of applied mathematics. “The selection of a best element (with regard to some criteria) from some set of available alternatives.” E.g. Travelling salesman (TSP).

E.g. algorithms: gradient-based methods, evolutionary algorithms, stochastic algorithms.

Interplanetary trajectories - space mission trajectories are defined by sets of parameters: launch date, initial velocity vector, sequence of flybys, sequence of deep-space manoeuvres (DSM).

Usually we want to minimise fuel consumption.

The resulting optimisation problem is hard: multimodal objective function, highly nonlinear, highly dimensional.

Traditionally tackled by teams of human experts.

Bio-inspired algorithms:

  • genetic algorithms
  • differential evolution
  • ant-colony optimisation
  • artificial bee-colony optimisation

Island model - name inspired by Darwin’s trip to the Galápagos Islands.

History of PaGMO - pattern of scientific programming code: created as part of some research and then abandoned for many years; not useable by anyone else and then picked up later and made more consumable by others.

Initially created Python bindings to initially created C/C++ ‘research’ code (2008-2009) and followed ‘eat own dog food’ approach by using it a lot for internal research. Been through 2 GSoCs and now ‘fully-fledged’ open source project.

Pros of Scientific Computing

  • Emphasis on correctness & reproducibility
  • Powerful driver for innovations in HPC

Cons of Scientific Computing

  • Wheel re-invention
  • Code is often written with a one-paper-horizon mindset
  • Most scientists are not trained in software engineering

PaGMO framework:

  • The abstract island class includes a problem, an algorithm and a population of candidate solutions, and a virtual evolve() method that dispatches the optimisation (to a thread or a process on another machine)

Implemented via Boost.Python.

Challenging, non-trivial issues:

  • serialisation across language boundaries involving virtual classes, base pointers, etc

  • extension from Python of C++ base classes

  • working around some of Python’s limitation wrt parallel programming (GIL)

Leveraging Python’s strengths:

  • Scientific Python stack: NumPy (crunching results), SciPy (optimisation algorithms), matplotlib, IPython, etc

PaGMO uses: evolution of neural networks for autonomous Martian rovers, selection of Near Earth Asteroids for future human missions.

Astropy - “a community package for Astronomers”:

  • Handle practical needs of astronomers: units, coordinates, FITS files, cosmological calculations

  • "One of best community packages ever seen"

  • Not research package like PaGMO

  • Heavy reliance on NumPy

Identifying birds, butterflies, and wildflowers with a snap

From Planet Plone. Published on Sep 13, 2014.

Identifying birds, butterflies, and wildflowers with a snap

Idenitfying birds, butterflies, and wildflowers with a snap

From Planet Plone. Published on Sep 13, 2014.

Idenitfying birds, butterflies, and wildflowers with a snap

Java for Python Developers - Chris Neugebauer

From Planet Plone. Published on Sep 12, 2014.

One company who likes Python but in reality likes Java a lot more = …. Google!

.java -> [compiler] -> JVM ByteCode -> JVM

Jython - setuptools - make .jars

from java.Lang.system

What can do in Python 2.7 can also do in Jython.

Android devices outsold full-size computers 2x.

Dalvik Virtual Machine = Android Runtime (ART)

.java -> [compiler] -> .dex (Dalvik exe) -> DVM

No dynamic 3rd-party libraries. All statically compiled into Dalvik exe.

Java Native Interface (JNI).

Kivy

Python-For-Android: Cpython API for Java.

from jnius import autoclass

PyJnius runs on Android!

JNI method signatures are a nightmare.

If Java’s more important, use Jython! Support for C extensions is not complete.

If Python’s more important, use PyJnius!

Jython is corporate sponsored by Sun.

2.7 support on Jython is only recent.

Kivy doesn’t have a very native look-and-feel. They say they’re working on this but “they don’t know what they’re talking about”.

Toga aims to use the native APIs for theming but currently has no Android support.

Intro to flask-security - Beau Butler

From Planet Plone. Published on Sep 12, 2014.

Why Flask? Steep Django learning curve. Like to start simple and then become more complex.

App example code: http://github.com/oddy/bangingminimal

Don’t get Async mail sending, Pretty forms.

You get: sess auth + RBAC, base users db, all std workflows, passwd crypto, token/JSON auth.

pip install flask flask-security flask-SQLAlchemy pyBcrypt paste

Google: Prove Domain ownership - use button option.

Black box testing: Know lots of hackers.

Need motivate hackers - apply drink.

How hacked? ARP Redir (had PC on same subnet), SSLSplit, click forgot password, intercept reset URL, reset URL.

Actual hack - other PC pretending to be Google SMTP server (on same subnet) intercepted program call and passed call on to Google SMTP server and hacker’s app.

Py smtplib doesn’t check Google’s cert! 2.x smtplib can’t do it at all.

Admin Ronacher FTW.

Flask-Sec RBAC itself still not broken.

Have a go: pwnoddy.com.

Packaging a Python desktop application using PyInstaller - Glenn Ramsey

From Planet Plone. Published on Sep 12, 2014.

Gathers the dependencies for your script and puts them all in a directory. Doesn’t actually create an installer and doesn’t include the source code.

Works on Windows, OSX, Linux.

Works with Python 2.4-2.7.

Very well documented.

Alts: Py2app - OSX, Py2exe - Windows.

Launches a boot loader then launches your script.

Can choose one file mode or one directory (default) mode.

On first run it will create a spec file. One file is better for smaller scripts as has to decompress file as part of ‘install’.

OSX - extra step that creates app bundle.

Can’t include files outside the app dir.

sys._MEIPASS: directory where app is installed.

OSX: need to tell it not no create icon on dock when running ‘installer’.

collect_data_files: collects CSVs.

Hidden imports: for imports that PyInstaller can’t find.

OSX: CMake/CPack. make package.

Windows: NSIS.

Example program: https://github.com/glennra/kpc2014

When debugging - run boot loader directly: CountryCode.app/Contents/MacOS/CountryCode

for b in a.binaries: print b

Python is slow, make it faster with C - Ben Shaw

From Planet Plone. Published on Sep 12, 2014.

C Module - traditional way. Write your functions in C (usually just wrapper code).

CTypes - newer way since 2006. Integration a bit more difficult. Great to use if don’t need Python types, exceptions, & objects back.

CDLL: reference to DLL

Init module that sets up the module.

Pypy: libraries written in C may not work. No need to change Python code.

Pure C: 11.2x faster, Pypy: 5.7x faster.

Slides: http://bbit.co.nz/blog/4/

External dependencies in web apps: system libs are not that scary - Francois Marier

From Planet Plone. Published on Sep 12, 2014.

Common example: 250 external dependencies, 8 different copies of same app, 4 different versions.

Have to patch same app 4x.

1. Only use Python libraries that are packaged for Debian.

2. Only use the version from the latest Debian release (stable).

Uses fabric to deploy libravatar app.

Result? Limited choice of Python libraries. Sometimes have to choose a less interesting library.

apt-cache search ^python = 2248

Cannot use the latest features. “Don’t know what’s in Django 1.7 so don’t miss anything”.

“Once every 1 or 2 years have to upgrade distro so not too bad and wasn’t too major”.

Optimising for sysadmins instead of developers.

Currently working on Vagrant script to help newcomers install it.

Non-minified jQuery. Have to minify in build step.

Cannot easily use unattended-upgrades. Django sometimes disables features as they’re fundamentally broken and when you do unattended-upgrades the feature you’re relying on suddenly is no longer available.

apticron - span you every day re. available upgrades until you upgrade.

Security updates not always timely in distros.

Only takes a single Debian user to file a security bug to start the process of getting it all fixed.

Is it realistic? After announcing talk a dev in Russia contacted Francois and said that is using exact same approach for SkyDNS.

Not your full-time job.

Uses a mature framework, e.g. NodeJS changing too quickly.

Consulting company, e.g. Catalyst IT. Small clients don’t like paying lots for system maintenance.

Distros abstract away all the language-specific incompatibilities.