How to think in Plone (when those  about you think in Drupal)

Dan Jacka

• Let’s make switching to Plone easier
• Plone content is naturally structured, drupal content has nodes indentified by numbers.
• For Plone lots of content is a prerequisite
• Confusions when coming from Drupal; content ‘lives’ where it appears and restrict locals adds with permission
• ‘micro components’ are powerful ways to add functionality
• Use micro components like dexterity behaviours, collective.contentleadimage, collective.watcherlist. Small products are good!
• Big products are bad, if you only need to use a part and get unnecessary stuff you don’t need.
• With Zope Component Architecture you can override specific parts in Plone, like the breadcrumbs 
• Store data; With context use zope annotations and archetypes.schemaextender. Without context use plone.app.registry, portal_properties and local utilities
• See:  http://www.martinaspeli.net/articles/dcworkflows-hidden-gems

ZODB tips and tricks

Carlos de la Guardia

• See collective.zodbbrowser Provides access to all objects and their attributes
• Eye is an external tool to browse the database, without having to install (in buildout)
• OMG POSKey error: first backup! Fire up debugger, see  http://plonechix.blogspot.com/2009/12/definitive-guide-to-poskeyerror.html
• Get rid of persistent utilities, see wildcard.fixpersistentutilities
• Always backup first before trying to fix anything!
• Restore data / do an undo from data that is deleted a long time ago. Use zc.beforestorage, a wrapper around storage to show site like it was on a certain date
• RelStorage, drop in replacement for file storage. Designed for high volume sites, multiple zodb instances can share the same database. Starts quickly regardless of db size. Supports undo, packing blobs. Capable of fail-over to replicated databases.
• Zc.zodbactivitylog (track db actvity), zodbshootout (benchmarks zeo vs relstorage), zodbupdate (helps you rename classes), dm.historical (get history of objects), dm.zodb.repair (restore lost objects)
• See Products.ZMIntrospection to look in objects
• Use PersistentDict for small amount of items, use OOBTree (and friends) for large amounts
• Use BTrees.length instead of len, much faster and avoids conflict errors
• Use zc.zlibstorage for dbs with lots of text, saves 60/70% of storage
• Use zc.zodbgc an inverse graph of db
• Increase the ZEO client cache size, when everything is crumbeling around you. When increasing round trips to ZODB are increased   

Carlos is writing a book on the zodb: http://zodb.readthedocs.org/en/latest/index.html 

Mistakes made and lessons learnt

Matt Hammilton And Matt Sital-Singh

• Use case a e-learning system, lot of content, users editing content but not necessary aware of it, lots of clustering of load on resources
•  30.000 user accounts, 160.000 messages, QA objects etc
• Don’t create  ghost content. If you need shared content, it can be a good idea to don’t use subsites but sync the content
• Use optimization products. See experimental namespace; queryplan, contentcreation, daterangeindexoptimisations, aggresiveopaquesetup, indexing. Most of them aren’t experimental anymore and are included in plone 4.2
• Lot’s of answers/questions in catalog. Up to 1.400.000 qa objects. Uhm sql? Or catalog multiplex tool, separate these objects from main catalog.
• Pre-load Catalog QueryPlan
• Keep things you don’t need out of the catalog
• If you know where something is – Don’t search for it. Hardcoding is ok
• Use unrestrictedSearchResults if possible, the catalog has a lot less work to do
• Maybe should not have used Plone. A lot of the data is relational.

Beginner mistakes. Expert failures

Alan Runyan

Beginners:

• Use logging/sending email or long term persistence for error logs
• Don’t mix dev/stage/prd concept/concerns
• Don’t use directlyProvides or using the ZMI to apply marker interfaces
• Do learn the profiler, see collective.stats (how many load/stores per request)
• Don’t attempt to ‘normalize’ the model
• Willing to claim defeat and backtrack; sometimes you simply do
• When beginning development use an alpha, beta or RC. If the project is finished, chances are big a stable version is released.
• Don’t be willing to take pain. Give feedback!

Experts:

• Using components early, not writing reusable code is ok.
• Overloading too much functionality in view
• Mounting / splitting ZODB in effort to make things faster
• Using Plone inappropriately. Security, workflows, staging/versioning, content, add-on components.
• See book scalability rules
• Unwilling to thorougly understand tradeoffs between ZODB/RDBMS
• Masking over performance problems with caching

Check out ploud and  ptah .

RelStorage is stable and is used in many production sites. It works with PostgreSQL, MySQL and Oracle. The rationale to use it is replication. SQL datbases can do this, the ZODB can’t.

Dextery in the wild

David Glick

• Case study of a complex Dexterity based solution, Net Impact
• Sales Force integration
• Membrane is used to represent users as content items. See dexterity.membrane
• Membrane the ugly: extra catalog with unneeded index
• Using dexterity with grok saves writing ZCML
• Content items are defined in (super) models
• Content items have multiple forms with different fields. Auto form can’t be used, dexterity.EditForm class view is used.
• See collective.z3cform.datagridfield, eea.facetednavigation, collective.salesforce.content  (dexterity behaviour)
• Dexterity content types is only used for custom content. Existing content type are modified using schema extender.

Tsunami proof Plone

Adam Terrey

• Building high available “web scale” Plone platform for emergency services
• Requirements; scale-ability, multisite, 99,99% reliability
• The cloud isn’t reliable enough
• CDNs are used for serving content
• Even the slightest amount of caching let’s Plone ‘breathe’ under heavy load
• Everything in the hosting stack must be redundant. Expensive!
• Datacenter should have multiple backbones
• DB redundancy is done with relstorage
• Funkload is used for stress testing

Progressive enhancement with wsgi 

Matthew Wilkes

• Wsgi is just an API for handeling http requests
• Use wsgi as middleware to add functionality to your application
• Good wsgi libraries: WebOb, makes requests easy to deal with / wsgirefWSGI web server is in the std lib
• The ZopeSkel equivilent for wsgi: wsgitemplates
• Example captcha’s, most form libraries have a different way to implement captcha’s
• Uses wsgi middleware to rewrite the form
• The backend application has a checkbox to see if the user is human. Middleware inserts a captcha widget. If theform is submitted the middleware checks the captcha input and on valid input the ’is human’ box is checked. On valid captcha input the form is stored, else an error is shown.
• Probably overkill, more initial effort but can be easy to apply to customer sites.
• Example for how to use wsgi middleware
• See islay.simplecaptcha and islay.hardercaptcha

Multiplayer Plone : Realtime collaboration 

Geir Bækholt

• Jabber/XMPP protocol is used, realtime and asyn
• Presence (status), message, iq (subscription),
• Communication not directly thru Plone, but via javascript
• Scales to large amount of users
• Collaborative writing like Google Docs in ATContent and dexterity
• Probably same diff match patch algorithm as in google docs
• Future: content notifications, async queues with xmpp in plone, conferencing, video and audio chat.
• See jarn.xmpp.buildout 

Killer feature/selling point for intranets and sites with lots of authors.

Clone to Plone

Adam Terrey

• How to move from older ‘systems’ to Plone. Migrating content, it’s pretty hard.
• Funnelweb http://plone.org/products/funnelweb
• A web crawler, uses mr.migrator and transmogrifyer
• Buildout, command line or Plone TTW

• Crawler is configurable; ignore regex, drop content using tal, amount of items, ignore robots meta
• Crawler saves in cache on disk
• Content extraction using xpath
• Restructure content, tidy titles
• Upload to Plone

• What’s not converted: dynamic content, front pages, collections, portlets.
• Diazo is used to create to look and feel of the old site
• Used to migrate big sites, est time to migrate a big site is 3 / 4 days

Making Plone Mobile using Responsive Web Design

Rob Porter

• Responsive design, geared towards lot’s of devices (and screen sizes!)
• Why design websites for mobile? Because there’s a huge increase on mobile device usage.
• See http://bostonglobe.com http://forefathersgroup.com and make the browser window smaller. The layout changes but content stays the same.
• See http://mediaqueri.es
• Define a range of screen resolutions with @media, to cater different devices. Each resolution range has an own layout/style/css.
• See plonetheme.responsivetheme, makes Plone responsive
• Book to buy Responsive Web design – Ethan Marcotte
• First start with smallest screen sizes and then go bigger

Creating websites using responsive desgn better than creating specific iDevice, Android, Blackberry apps.

The future search of Plone

Sally Kleinfelt, Hanno Schlichting and someone from six feet up

• Information retrieval
• Two search engines ZCatalog and Solr
•  ZCTextIndex is very simple, TextIndexNG adds multilingual, better parsing, binary transforms, synonyms.
• Solr, based on Lucene Java search library. RESTful APIs. Powers twitter and wikipedia. An army of engineers working on it
• Solr has way more search features than ZCatalog
• Collective.solr and alm.solrindex are available add-ons. Collective.recipe.solrinstance as buildout recipe
• Solr indexing is not done transactional, it takes a while before the contt is indexed

De PyGrunn ‘Python and friends’ conferentie is gehouden in Het Paleis in Groningen en is bezocht door ruim 80 personen. Voorafgaand aan de conferentie was er een masterclass van Armin Ronacher. De conferentie is gericht op de Python programmeertaal en verwante technologie. Met sprekers uit Nederland en het buitenland was de Python-community goed vertegenwoordigd op PyGrunn. Vanuit Goldmund, Wyldebeast & Wunderliebe heeft Kim Chee Leong een presentatie gegeven over buildout, dit is een tool voor het uitrollen van applicaties.

Goldmund, Wyldebeast & Wunderliebe en Paylogic hebben de conferentie georganiseerd. Vanwege het grote success zal PyGrunn er ook volgend jaar weer zijn.

This example is applicable to Plone 3 and 4. But the jQuery/Maps API are generic, so if you’re not interested in the Plone stuff just skip to the jQuery code.

Use case:

• Select a location based on an address or by dragging a marker in the map. Store latitude, longitude and zoom level.
• Display a Google Map widget on your content type that show a stored location

Prequisites

• You know how to create your custom content type in Plone (if you want to integrate this example in Plone).
• Basic javascript/jQuery skills

Getting started

Your custom content type

We want to store the longitude, latitude and zoom level in the content.

Create you own custom content types and add the following fields:

• longitude and latitude in a string field
• zoomLevel in a integer field

Include necessary javascript and CSS

We are including jQuery. jQuery UI with only the live search function included, this allows direct results when searching for a location. The Google Maps API v3. And some CSS for the map canvas and the live search field.

If you’re planning to use the jquery libraries from my site, please download the files and host these on your own site.

<script src="http://www.leong.nl/wp-content/jquery-1.4.2.min_.js" type="text/javascript"><!--mce:0--></script>
 
<script src="http://www.leong.nl/wp-content/jquery-ui-1.8.1.custom.min_.js" type="text/javascript"><!--mce:1--></script>
 
<script src="http://maps.google.com/maps/api/js?sensor=false" type="text/javascript"><!--mce:2--></script>
 
<style type="text/css">
#map_canvas {        
  width: 650px;    
  height: 500px;        
  margin-bottom: 2em;  
} 
 
.ui-autocomplete  {        
  background-color: white;       
  width: 300px;        
  border: 1px solid #cfcfcf;        
  list-style-type: none;        
  padding-left: 0px;        
  cursor: pointer; 
}  
</style>

The Google Maps jQuery code

This is jQuery code for displaying and selecting locations in Google Maps. Most of the Maps API code used in this example comes from this article on cibul.org. The article has a good explanation how the Maps API works.

Short explanation on the code below. The inserting of the Maps and search field is all done in javascript code. Without this code you can still save or view the lat/long coordinates. Based on editing or viewing content different functions are called.

 
// based on: http://tech.cibul.org/geocode-with-google-maps-api-v3/
// replace jq with $ character if you're using this code outside Plone
 
var DEFAULT_LATITUDE = 53.21310;
var DEFAULT_LONGITUDE = 5.71713;
var DEFAULT_ZOOM_LEVEL = 9;
 
var ploneMaps = {
 
    /**
     *  Show the map based on saved location
     */
    googleMapsView: function(message) {
        jq('#archetypes-fieldname-latitude').hide();
        jq('#archetypes-fieldname-longitude').hide();
        jq('#archetypes-fieldname-zoomLevel').hide();
 
        var html = "<label>Location:</label><br/>";
        html += "<div id='map_canvas'></div>";
        jq(html).insertBefore('#archetypes-fieldname-latitude');
 
        var longitude = parseFloat(jq("#parent-fieldname-longitude").text());
        var latitude = parseFloat(jq("#parent-fieldname-latitude").text());
        var zoom = parseInt(jq("#parent-fieldname-zoomLevel").text());
 
        ploneMaps.googleMapsDraw(longitude, latitude, zoom, true, false);
    },
    /**
     *  Show the saved location or starting point.
     *  Set-up a search widget for address geocoding
     */
    googleMapsEdit: function(message) {
        jq('#archetypes-fieldname-latitude').hide();
        jq('#archetypes-fieldname-longitude').hide();
        jq('#archetypes-fieldname-zoomLevel').hide();
 
        // create search field
        var html = "<div class='field' id='google-map-search'>";
        html += "<label for='map-search'>Location</label>";
        html += "<div class='formHelp'>Use the marker to select a location ";
        html += "or search field to select a location.</div>";
        html += "<input name='map-search' type='text' size='45'></input>";
        html += "<div id='map_canvas'></div>";
        html += "</div>";
 
        jq(html).insertBefore('#archetypes-fieldname-latitude');
 
        // no form submit when an enter is given in search field
        jq('#google-map-search').keypress(function(e){
           if(e.which == 13){
               e.preventDefault();
           }
        });
 
        // save zoomlevel on submit
        jq('form#plaats-base-edit').submit(function(e){
           jq("#zoomLevel").val(map.getZoom());
        });
 
        // saved location
        var latitude = jq('#latitude').val();
        var longitude = jq('#longitude').val();
        var zoomLevel = parseInt(jq('#zoomLevel').val());
 
        // use default location if nothing is saved
        if (!latitude || !longitude) {
            latitude = DEFAULT_LATITUDE;
            longitude = DEFAULT_LONGITUDE;
            zoomLevel = DEFAULT_ZOOM_LEVEL;
        }
 
        var data = ploneMaps.googleMapsDraw(longitude, latitude, zoomLevel, true, true);
        var map = data[0];
        var markerObj = data[1];
 
        ploneMaps.googleMapsHelper(map, markerObj);
    },
    /**
     * Draw the location based on specific parameters
     */
    googleMapsDraw: function(longitude, latitude, zoomLevel, marker, drag) {
        var map = false;
        if (!isNaN(longitude) && !isNaN(latitude)) {
            var location = new google.maps.LatLng(latitude, longitude);
 
            var myOptions = {
              zoom: zoomLevel,
              center: location,
              mapTypeId: google.maps.MapTypeId.ROADMAP
            };
            map = new google.maps.Map(document.getElementById("map_canvas"), myOptions);
            if (marker) {
                marker  = new google.maps.Marker({position: location, map: map, draggable: drag});
            }
 
        }
        return [map, marker]
    },
    /**
     * Helper function, ajax address search and marker event listener
     */
    googleMapsHelper: function(map, marker) {
        //GEOCODER
        geocoder = new google.maps.Geocoder();
        // Form auto completion
        jq("input[name='map-search']").autocomplete({ 
            //This bit uses the geocoder to fetch address values
            source: function(request, response) {
                geocoder.geocode( {'address': request.term, 'region': 'NL' }, function(results, status) {
                  response(jq.map(results, function(item) {
                    return {
                      label: item.formatted_address,
                      value: item.formatted_address,
                      latitude: item.geometry.location.lat(),
                      longitude: item.geometry.location.lng()
                    }
                  }));
                })
            },
            // This bit is executed upon selection of an address
            select: function(event, ui) {
                jq("#latitude").val(ui.item.latitude);
                jq("#longitude").val(ui.item.longitude);
                var location = new google.maps.LatLng(ui.item.latitude, ui.item.longitude);
                marker.setPosition(location);
                map.setCenter(location);
                map.setZoom(11);
            }
        });
        // Reverse Geocoding, when the marker is being dragged on the map update the fields.
        google.maps.event.addListener(marker, 'drag',  function() {
            geocoder.geocode({'latLng': marker.getPosition()}, function(results, status) {
              if (status == google.maps.GeocoderStatus.OK) {
                if (results[0]) {
                  jq("input[name='map-search']").val(results[0].formatted_address);
                  jq("#latitude").val(marker.getPosition().lat());
                  jq("#longitude").val(marker.getPosition().lng());
                }
              }
            });
        });
    }
}
 
jq(document).ready(function() {
 
    // match this jquery selector on the view template of your content type
    if (jq('body.template-plaats_view.portaltype-plaats').length >= 1) {
        ploneMaps.googleMapsView();
    }
    // match this jquery selector on the edit template of your content type
    else if (jq('body.template-base_edit.portaltype-plaats').length >= 1) {
        ploneMaps.googleMapsEdit();
    }
 
});

Update 25 May 2010 – This information is outdated. Please refer to this page on the XDA Developers forum:

http://forum.xda-developers.com/showthread.php?t=529062

Here are the steps to the how-to’s to get root access to your HTC Magic and load a custom ROM. I took me some time to find all the needed info to do the job. So I’m hapy to share it. There’s a lot of (cluttered) info but most of it is found in multiple page forums. Really great that there lot’s of people developing and using this stuff but a forum isn’t the right place for documentation.

Note 30-03-2010, the information below is outdated. Please look at the following wiki’s for more up to date info:

http://wiki.xda-developers.com/index.php?pagename=HTC_Sapphire_Hacking

http://wiki.cyanogenmod.com/index.php/Main_Page

Got root?

Getting root access on the Magic isn’t so hard. Just install the SDK and USB drivers and push the images to the phone. In the last step is to install haykuro’s SPL update. This is the bootloader (correct?) and has a very usefull back-up option. It’s called nandroid and creates back-up images of the system. After this you have root access from the android debugger on your computer.

http://android-dls.com/wiki/index.php?title=Magic_Rooting

The second step is enabling root access from the phone. This allows you to su from the phone. Needed for installing rooted apps. It opens a security risk as mentioned in the how-to. But there a sudo-like app SuperUser whitelist to prevent unwanted root access (preinstalled on most custom roms). :

http://android-dls.com/wiki/index.php?title=Magic_Root_Access

Custom ROMS

So now you’ve got full access to the Magic but stuck with a stock rom from your provider. There are lots of roms floating around for Android but most of them are for the HTC Dream (G1). I found out the hard way by installing a G1 rom, WiFi and other hardware functions aren’t working. Not so strange with a different kernel etc…

These are the ROMs that are available at the moment:

• Google/HTC original stock rom
• Hero Rom. Android with an updated ui. Shiney and new but it’s slow and laggy
• Google ION rom. ION is a developer phone. Customized rom available.
• Haykuro’s Rom. Haykuro’s roms are excellent but it seems that he has stopped developing.
• Smartphone France (translation). Customized ION rom, available in EU languages.

Sending a rom is really easy, just like in the rooting process with an update.zip on the SD card. Always take care when installing a new rom. Better safe than sorry and check if it works on your Magic/Sapphire/ION. The last thing you want is a bricked phone…

I tried the Google ION rom and it works well. The benefits of this custom rom are; it’s faster, rooted, voice commands and more!� This is a good rom but there’s a HTC soft keyboard instead of Google’s. I didn’t like it and replaced it. Instructions here, you’ll need a specific rom (for extracting system files) or it won’t work!

I’m now using the Smartphone France version. The main advantage between above ION rom is this one is, tethering works and a higher version update rate. You won’t notice anything of the French language (beside a few small apps on the rom).

Also check out my Android bookmarks!

If you want try buildout for Django try this how-to.

Buildout is well known in the Zope/Plone world and can be used in both development and production stages. Because a production mod_python/Django setup is a bit different them I’m used to (using Zope behind Apache), I decided to use buildout to generate a Apache virtual host config.

You can follow these these steps on this wiki for a buildout with mod_python:

https://tracpub.yaco.es/djangobuildout/ (warning! just click thru the SSL errors)