Software Engineering

Software Engineering

1635 bookmarks
Newest
Getting Clean With PHP | Nettuts+
Getting Clean With PHP | Nettuts+
Data security is important and often undervalued by designers, developers, and clients alike. Since PHP 5.2.0, data sanitization and validation has been made significantly easier with the introduction of data filtering. Today, we’re going to take a closer look at these filters, how to use them, and build a few custom functions.
Getting Clean With PHP | Nettuts+
Deltacloud | Many Clouds. One API. No Problem.
Deltacloud | Many Clouds. One API. No Problem.

Start an instance on an internal cloud, then with the same code start another on EC2 or Rackspace. Deltacloud protects your apps from cloud API changes and incompatibilities, so you can concentrate on managing cloud instances the way you want.

Deltacloud Core gives you:

REST API (simple, any-platform access) Support for all major cloud service providers Backward compatibility across versions, providing long-term stability for scripts, tools and applications One level up, Deltacloud Aggregator provides a web UI in front of the Deltacloud API. With Deltacloud Aggregator, your users can:

View image status and stats across clouds, all in one place Migrate instances from one cloud to another Manage images locally and provision them on any cloud

Deltacloud | Many Clouds. One API. No Problem.
High Scalability - High Scalability - Product: Scribe - Facebook's Scalable Logging System
High Scalability - High Scalability - Product: Scribe - Facebook's Scalable Logging System

Client Code Interface - How does your code interact with the log system? Scribe doesn't do much for you here. There's a simple Thrift interface for logging from a large set of languages, but the bulk of the work is stull up to you. Distribution System - This is were Scribe fits. It reliably (mostly) moves large numbers of messages around. A few error cases lead to data loss: 1) If a client can't connect to either the local or central scribe server the message will be loss; 2) If a scribe server crashes it could lose a small amount of data that's in memory but not on disk; 3) Some multiple component failure cases, such as a resender can't connect to any central server and its local disk fills up; 4) Some rare timeout conditions can lead to duplicate messages Do Something Usefullizer - How do you do anything useful with over 1 million messages per second? Good question. Scribe doesn't help here. But Scribe will get your data their.

High Scalability - High Scalability - Product: Scribe - Facebook's Scalable Logging System
Welcome to Pig!
Welcome to Pig!

Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.

At the present time, Pig's infrastructure layer consists of a compiler that produces sequences of Map-Reduce programs, for which large-scale parallel implementations already exist (e.g., the Hadoop subproject). Pig's language layer currently consists of a textual language called Pig Latin, which has the following key properties:

Ease of programming. It is trivial to achieve parallel execution of simple, "embarrassingly parallel" data analysis tasks. Complex tasks comprised of multiple interrelated data transformations are explicitly encoded as data flow sequences, making them easy to write, understand, and maintain. Optimization opportunities. The way in which tasks are encoded permits the system to optimize their execution automatically, allowing the user to focus on semantics rather than efficiency. Extensibility. Users can create their own functions to do special-purpose processing.

Welcome to Pig!
What Is This Devops Thing, Anyway?
What Is This Devops Thing, Anyway?
In the last few months, a movement has begun to take shape. It's a movement of people who think it's time for change in the IT industry - time to stop wasting money, time to start delivering great software, and building systems that scale and last. This movement is being called Devops. But what is Devops? Where did it come from? And what can it achieve?
What Is This Devops Thing, Anyway?
Do you have a DevOps Culture? — Agile Web Operations
Do you have a DevOps Culture? — Agile Web Operations
A lot of Sysadmins and developers all over the world write, meet and talk about DevOps: How to collaborate better so we can deliver business value faster. The aim of DevOps is to get rid of the traditional way of thinking in silos inherent to development and operations. But how can You find out whether your organization is ready for DevOps? Here are a few hints.
Do you have a DevOps Culture? — Agile Web Operations
Test your Facebook application | FriendRunner
Test your Facebook application | FriendRunner

One of the great difficulties of Facebook application development is the difficulty in running large-scale tests due to the large number of Facebook accounts required, and the Facebook Terms of Use which does not allow automated systems to access the Facebook servers. These issues are explained in depth in Load testing Facebook apps (Part 1) and Load testing Facebook apps (Part 2). FriendRunner acts as a drop-in replacement for the Facebook servers, and so allows you to run your tests with as many users as you'd like. Furthermore, FriendRunner's automated test system runs the entire test for you once you create a test script to define the actions for a single user.

Test your Facebook application | FriendRunner
High Scalability - High Scalability - Scaling an AWS infrastructure - Tools and Patterns
High Scalability - High Scalability - Scaling an AWS infrastructure - Tools and Patterns
How do you scale an AWS (Amazon Web Services) infrastructure? This article will give you a detailed reply in two parts: the tools you can use to make the most of Amazon’s dynamic approach, and the architectural model you should adopt for a scalable infrastructure.
High Scalability - High Scalability - Scaling an AWS infrastructure - Tools and Patterns
MySQLnd Plugins: Writing a MySQL Query Logger in PHP - ThinkPHP /dev/blog
MySQLnd Plugins: Writing a MySQL Query Logger in PHP - ThinkPHP /dev/blog
A new approach to implementing a query logger and potentially more complex features such as monitoring or read/write-splitting is the MySQLnd Userland Handler Extension (mysqlnd_uh, pecl website). The extension lets you register a PHP class as a proxy for every MySQLndconnection. Every call to a function to MySQLnd (usually indirect through mysqli, mysql, pdo_mysql) is passed to the PHP class, which then calls the original MySQLnd function. The extension makes it possible to use a custom userland class as a transparent proxy for all MySQLnd frontend (again mysqli, mysql, pdo_myslq). In the next section, I'll outline the requirements of mysqlnd_uh and show a sample implementation.
MySQLnd Plugins: Writing a MySQL Query Logger in PHP - ThinkPHP /dev/blog
relevance's cap_gun at master - GitHub
relevance's cap_gun at master - GitHub

Tell everyone about your releases! Send email notification after Capistrano deployments! Rule the world!

Drop your ActionMailer configuration information into your deploy.rb file, configure recipients for the deployment notifications, and setup the callback task.

Setup and configuration are done entirely inside your deploy.rb file to keep it super simple. Your emails are sent locally from the box performing the deployment, but CapGun queries the server to grab the necessary release info.

This even includes the Net::SMTP TLS hack inside as a vendored dependancy to allow super easy email sending without setting up an MTA.

relevance's cap_gun at master - GitHub
orieg's php-judy at master - GitHub
orieg's php-judy at master - GitHub

php-judy is an exentsion by Nicolas Brousse for the Judy C library.

http://github.com/orieg/php-judy

A Judy array is a complex but very fast associative array data structure for storing and looking up values using integer or string keys. Unlike normal arrays, Judy arrays may be sparse; that is, they may have large ranges of unassigned indices.

http://en.wikipedia.org/wiki/Judy_array

Judy is a C library that implements a dynamic array. Empty Judy arrays are declared with null pointers. A Judy array consumes memory only when populated yet can grow to take advantage of all available memory. Judy's key benefits are: scalability, performance, memory efficiency, and ease of use. Judy arrays are designed to grow without tuning into the peta-element range, scaling near O(log-base-256) -- 1 more RAM access at 256 X population.

http://judy.sourceforge.net
orieg's php-judy at master - GitHub
Krati
Krati
Krati is a simple persistent content data store with very low latency and high throughput. It is designed for easy integration with other services with little effort in tuning configuration, performance and JVM garbage collection. This software is published under the terms of the Apache Software License version 2.0, a copy of which has been included in the LICENSE file shipped with the Krati distribution.
Krati
Designing Web Applications for Scalability
Designing Web Applications for Scalability
In this post, I'm going to go over some of the strategies I've used to architect web applications for scalability, right from the start of the design process, in such a way that I'm prepared to scale when I need to, but not forced into doing so before its necessary. Easing the transition from small scale to large scale can be made much easier by choosing the right technologies and implementing the right coding patterns up front.
Designing Web Applications for Scalability
InfoQ: Your Scrum Checklist: Scrum Hard Facts: Roles. Artefacts. All Meetings
InfoQ: Your Scrum Checklist: Scrum Hard Facts: Roles. Artefacts. All Meetings

Scrum is a framework with simple rules. This Scrum Checklist will help you to remember these simple rules in the heat of daily work and stress. It enable you to create an enjoyable and productive work environment with your Scrum-Team.

For Scrum Beginners - Follow the Checklists, do Scrum by the book.This will enable you to run you first 3 to 4 Sprints very successfully. Your success will facilitate to spread Scrum in your organization.

For Scrum Intermediates - Use your common sense to tweak your processes guided by these Scrum Checklists.

As an experienced ScrumMaster - Use Scrum Check- lists as your safety belt in stressful situations.

Scrum Checklists do not replace experience and practice. Scrum Checklists are not procedures that you have to follow. Scrum Checklists help to run a successful Scrum in a demanding environment.

InfoQ: Your Scrum Checklist: Scrum Hard Facts: Roles. Artefacts. All Meetings
Leverage Points: Places to Intervene in a System | Solutions
Leverage Points: Places to Intervene in a System | Solutions

How do we change the structure of systems to produce more of what we want and less of that which is undesirable? After years of working with corporations on their systems problems, MIT’s Jay Forrester likes to say that the average manager can define the current problem very cogently, identify the system structure that leads to the problem, and guess with great accuracy where to look for leverage points—places in the system where a small change could lead to a large shift in behavior.

Leverage Points: Places to Intervene in a System | Solutions
Ibuildings - Horizontally Scalable Web Applications - ibuildings.com
Ibuildings - Horizontally Scalable Web Applications - ibuildings.com
What Is Scalability? Scalability is not high performanceWhat goes up, must come downScalable by Design Loose CouplingCreate APIs, assemble applicationsScaling Your Application Horizontal vs. vertical scalingBreaking It Apart – Thinking in Pieces StaticDynamic Database Session ClusteringMemcached Session ClusteringAPIsScaling Your Database Database ReplicationDatabase FederationDatabase Sharding Alternative Data Storage Engines API-only data accessCouchDB and Amazon SimpleDbScaling to the Cloud Platform as a ServiceInfrastructure as a ServiceDegrade gracefully
Ibuildings - Horizontally Scalable Web Applications - ibuildings.com