Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials - Programming

1083 Articles
article-image-microsoft-build-2019-microsoft-showcases-new-updates-to-ms-365-platfrom-with-focus-on-ai-and-developer-productivity
Sugandha Lahoti
07 May 2019
10 min read
Save for later

Microsoft Build 2019: Microsoft showcases new updates to MS 365 platform with focus on AI and developer productivity

Sugandha Lahoti
07 May 2019
10 min read
At the ongoing Microsoft Build 2019 conference, Microsoft has announced a ton of new features and tool releases with a focus on innovation using AI and mixed reality with the intelligent cloud and the intelligent edge. In his opening keynote, Microsoft CEO Satya Nadella outlined the company’s vision and developer opportunity across Microsoft Azure, Microsoft Dynamics 365 and IoT Platform, Microsoft 365, and Microsoft Gaming. “As computing becomes embedded in every aspect of our lives, the choices developers make will define the world we live in,” said Satya Nadella, CEO, Microsoft. “Microsoft is committed to providing developers with trusted tools and platforms spanning every layer of the modern technology stack to build magical experiences that create new opportunity for everyone.” https://youtu.be/rIJRFHDr1QE Increasing developer productivity in Microsoft 365 platform Microsoft Graph data connect Microsoft Graphs are now powered with data connectivity, a service that combines analytics data from the Microsoft Graph with customers’ business data. Microsoft Graph data connect will provide Office 365 data and Microsoft Azure resources to users via a toolset. The migration pipelines are deployed and managed through Azure Data Factory. Microsoft Graph data connect can be used to create new apps shared within enterprises or externally in the Microsoft Azure Marketplace. It is generally available as a feature in Workplace Analytics and also as a standalone SKU for ISVs. More information here. Microsoft Search Microsoft Search works as a unified search experience across all Microsoft apps-  Office, Outlook, SharePoint, OneDrive, Bing and Windows. It applies AI technology from Bing and deep personalized insights surfaced by the Microsoft Graph to personalized searches. Other features included in Microsoft Search are: Search box displacement Zero query typing and key-phrase suggestion feature Query history feature, and personal search query history Administrator access to the history of popular searches for their organizations, but not to search history for individual users Files/people/site/bookmark suggestions Microsoft Search will begin publicly rolling out to all Microsoft 365 and Office 365 commercial subscriptions worldwide at the end of May. Read more on MS Search here. Fluid Framework As the name suggests Microsoft's newly launched Fluid framework allows seamless editing and collaboration between different applications. Essentially, it is a web-based platform and componentized document model that allows users to, for example, edit a document in an application like Word and then share a table from that document in Microsoft Teams (or even a third-party application) with real-time syncing. Microsoft says Fluid can translate text, fetch content, suggest edits, perform compliance checks, and more. The company will launch the software developer kit and the first experiences powered by the Fluid Framework later this year on Microsoft Word, Teams, and Outlook. Read more about Fluid framework here. Microsoft Edge new features Microsoft Build 2019 paved way for a bundle of new features to Microsoft’s flagship web browser, Microsoft Edge. New features include: Internet Explorer mode: This mode integrates Internet Explorer directly into the new Microsoft Edge via a new tab. This allows businesses to run legacy Internet Explorer-based apps in a modern browser. Privacy Tools: Additional privacy controls which allow customers to choose from 3 levels of privacy in Microsoft Edge—Unrestricted, Balanced, and Strict. These options limit third parties to track users across the web.  “Unrestricted” allows all third-party trackers to work on the browser. “Balanced” prevents third-party trackers from sites the user has not visited before. And “Strict” blocks all third-party trackers. Collections: Collections allows users to collect, organize, share and export content more efficiently and with Office integration. Microsoft is also migrating Edge as a whole over to Chromium. This will make Edge easier to develop for by third parties. For more details, visit Microsoft’s developer blog. New toolkit enhancements in Microsoft 365 Platform Windows Terminal Windows Terminal is Microsoft’s new application for Windows command-line users. Top features include: User interface with emoji-rich fonts and graphics-processing-unit-accelerated text rendering Multiple tab support and theming and customization features Powerful command-line user experience for users of PowerShell, Cmd, Windows Subsystem for Linux (WSL) and all forms of command-line application Windows Terminal will arrive in mid-June and will be delivered via the Microsoft Store in Windows 10. Read more here. React Native for Windows Microsoft announced a new open-source project for React Native developers at Microsoft Build 2019. Developers who prefer to use the React/web ecosystem to write user-experience components can now leverage those skills and components on Windows by using “React Native for Windows” implementation. React for Windows is under the MIT License and will allow developers to target any Windows 10 device, including PCs, tablets, Xbox, mixed reality devices and more. The project is being developed on GitHub and is available for developers to test. More mature releases will follow soon. Windows Subsystem for Linux 2 Microsoft rolled out a new architecture for Windows Subsystem for Linux: WSL 2 at the MSBuild 2019. Microsoft will also be shipping a fully open-source Linux kernel with Windows specially tuned for WSL 2. New features include massive file system performance increases (twice as much speed for file-system heavy operations, such as Node Package Manager install). WSL also supports running Linux Docker containers. The next generation of WSL arrives for Insiders in mid-June. More information here. New releases in multiple Developer Tools .NET 5 arrives in 2020 .NET 5 is the next major version of the .NET Platform which will be available in 2020. .NET 5 will have all .NET Core features as well as more additions: One Base Class Library containing APIs for building any type of application More choice on runtime experiences Java interoperability will be available on all platforms. Objective-C and Swift interoperability will be supported on multiple operating systems .NET 5 will provide both Just-in-Time (JIT) and Ahead-of-Time (AOT) compilation models to support multiple compute and device scenarios. .NET 5 also will offer one unified toolchain supported by new SDK project types as well as a flexible deployment model (side-by-side and self-contained EXEs) Detailed information here. ML.NET 1.0 ML.NET is Microsoft’s open-source and cross-platform framework that runs on Windows, Linux, and macOS and makes machine learning accessible for .NET developers. Its new version, ML.NET 1.0, was released at the Microsoft Build Conference 2019 yesterday. Some new features in this release are: Automated Machine Learning Preview: Transforms input data by selecting the best performing ML algorithm with the right settings. AutoML support in ML.NET is in preview and currently supports Regression and Classification ML tasks. ML.NET Model Builder Preview: Model Builder is a simple UI tool for developers which uses AutoML to build ML models. It also generates model training and model consumption code for the best performing model. ML.NET CLI Preview: ML.NET CLI is a dotnet tool which generates ML.NET Models using AutoML and ML.NET. The ML.NET CLI quickly iterates through a dataset for a specific ML Task and produces the best model. Visual Studio IntelliCode, Microsoft’s tool for AI-assisted coding Visual Studio IntelliCode, Microsoft’s AI-assisted coding is now generally available. It is essentially an enhanced IntelliSense, Microsoft’s extremely popular code completion tool. Intellicode is trained by using the code of thousands of open-source projects from GitHub that have at least 100 stars. It is available for C# and XAML for Visual Studio and Java, JavaScript, TypeScript, and Python for Visual Studio Code. IntelliCode also is included by default in Visual Studio 2019, starting in version 16.1 Preview 2. Additional capabilities, such as custom models, remain in public preview. Visual Studio 2019 version 16.1 Preview 2 Visual Studio 2019 version 16.1 Preview 2 release includes IntelliCode and the GitHub extensions by default. It also brings out of preview the Time Travel Debugging feature introduced with version 16.0. Also includes multiple performances and productivity improvements for .NET and C++ developers. Gaming and Mixed Reality Minecraft AR game for mobile devices At the end of Microsoft’s Build 2019 keynote yesterday, Microsoft teased a new Minecraft game in augmented reality, running on a phone. The teaser notes that more information will be coming on May 17th, the 10-year anniversary of Minecraft. https://www.youtube.com/watch?v=UiX0dVXiGa8 HoloLens 2 Development Edition and unreal engine support The HoloLens 2 Development Edition includes a HoloLens 2 device, $500 in Azure credits and three-months free trials of Unity Pro and Unity PiXYZ Plugin for CAD data, starting at $3,500 or as low as $99 per month. The HoloLens 2 Development Edition will be available for preorder soon and will ship later this year. Unreal Engine support for streaming and native platform integration will be available for HoloLens 2 by the end of May. Intelligent Edge and IoT Azure IoT Central new features Microsoft Build 2019 also featured new additions to Azure IoT Central, an IoT software-as-a-service solution. Better rules processing and customs rules with services like Azure Functions or Azure Stream Analytics Multiple dashboards and data visualization options for different types of users Inbound and outbound data connectors, so that operators can integrate with   systems Ability to add custom branding and operator resources to an IoT Central application with new white labeling options New Azure IoT Central features are available for customer trials. IoT Plug and Play IoT Plug and Play is a new, open modeling language to connect IoT devices to the cloud seamlessly without developers having to write a single line of embedded code. IoT Plug and Play also enable device manufacturers to build smarter IoT devices that just work with the cloud. Cloud developers will be able to find IoT Plug and Play enabled devices in Microsoft’s Azure IoT Device Catalog. The first device partners include Compal, Kyocera, and STMicroelectronics, among others. Azure Maps Mobility Service Azure Maps Mobility Service is a new API which provides real-time public transit information, including nearby stops, routes and trip intelligence. This API also will provide transit services to help with city planning, logistics, and transportation. Azure Maps Mobility Service will be in public preview in June. Read more about Azure Maps Mobility Service here. KEDA: Kubernetes-based event-driven autoscaling Microsoft and Red Hat collaborated to create KEDA, which is an open-sourced project that supports the deployment of serverless, event-driven containers on Kubernetes. It can be used in any Kubernetes environment — in any public/private cloud or on-premises such as Azure Kubernetes Service (AKS) and Red Hat OpenShift. KEDA has support for built-in triggers to respond to events happening in other services or components. This allows the container to consume events directly from the source, instead of routing through HTTP. KEDA also presents a new hosting option for Azure Functions that can be deployed as a container in Kubernetes clusters. Securing elections and political campaigns ElectionGuard SDK and Microsoft 365 for Campaigns ElectionGuard, is a free open-source software development kit (SDK) as an extension of Microsoft’s Defending Democracy Program to enable end-to-end verifiability and improved risk-limiting audit capabilities for elections in voting systems. Microsoft365 for Campaigns provides security capabilities of Microsoft 365 Business to political parties and individual candidates. More details here. Microsoft Build is in its 6th year and will continue till 8th May. The conference hosts over 6,000 attendees with early 500 student-age developers and over 2,600 customers and partners in attendance. Watch it live here! Microsoft introduces Remote Development extensions to make remote development easier on VS Code Docker announces a collaboration with Microsoft’s .NET at DockerCon 2019 How Visual Studio Code can help bridge the gap between full-stack development and DevOps [Sponsered by Microsoft]
Read more
  • 0
  • 0
  • 37033

article-image-handle-odoo-application-data-with-orm-api-tutorial
Sugandha Lahoti
19 Jul 2018
17 min read
Save for later

Handle Odoo application data with ORM API [Tutorial]

Sugandha Lahoti
19 Jul 2018
17 min read
The ORM API, allows you to write complex logic and wizards to provide a rich user interaction for your apps. The ORM provides few methods to programmatically interact with the Odoo data model and the data, called the Application Programming Interface (API). These start with the basic CRUD (create, read, update, delete) operations, but also include other operations, such as data export and import, or utility functions to aid the user interface and experience. It also provides some decorators which allow us, when adding new methods, to let the ORM know how they should be handled. In this article, we will learn how to use the most important API methods available for any Odoo Model, and the available API decorators to be used in our custom methods, depending on their purpose. We will also explore the API offered by the Discuss app since it provides the message and notification features for Odoo. The article is an excerpt from the book Odoo 11 Development Essentials - Third Edition, by Daniel Reis. All the code files in this post are available on Github. We will start by having a closer look at the API decorators. Understanding the ORM decorators ORM decorators are important for the ORM, and allow it to give those methods specific uses. Let's see the ORM decorators we have available, and when each should be used. Record handling decorators Most of the time, we want a custom method to perform some actions on a recordset. For this, we should use @api.multi, and in that case, the self argument will be the recordset to work with. The method's logic will usually include a for loop iterating on it. This is surely the most frequently used decorator. If no decorator is used on a model method, it will default to  @api.multi behavior. In some cases, the method is prepared to work with a single record (a singleton). Here we could use the  @api.one decorator, but this is not advised because for Version 9.0 it was announced it would be deprecated and may be removed in the future. Instead, we should use @api.multi and add to the method code a line with self.ensure_one(), to ensure it is a singleton as expected. Despite being deprecated, the @api.one decorator is still supported. So it's worth knowing that it wraps the decorated method, doing the for-loop iteration to feed it one record at a time. So, in an @api.one decorated method, self is guaranteed to be a singleton. The return values of each individual method call are aggregated as a list and then returned. The return value of @api.one can be tricky: it returns a list, not the data structure returned by the actual method. For example, if the method code returns a dict, the actual return value is a list of dict values. This misleading behavior was the main reason the method was deprecated. In some cases, the method is expected to work at the class level, and not on particular records. In some object-oriented languages this would be called a static method. These class-level static methods should be decorated with @api.model. In these cases, self should be used as a reference for the model, without expecting it to contain actual records. Methods decorated with @api.model cannot be used with user interface buttons. In those cases, @api.multi should be used instead. Specific purpose decorators A few other decorators have more specific purposes and are to be used together with the decorators described earlier: @api.depends(fld1,...) is used for computed field functions, to identify on what changes the (re)calculation should be triggered. It must set values on the computed fields, otherwise it will error. @api.constrains(fld1,...) is used for validation functions, and performs checks for when any of the mentioned fields are changed. It should not write changes in the data. If the checks fail, an exception should be raised. @api.onchange(fld1,...) is used in the user interface, to automatically change some field values when other fields are changed. The self argument is a singleton with the current form data, and the method should set values on it for the changes that should happen in the form. It doesn't actually write to database records, instead it provides information to change the data in the UI form. When using the preceding decorators, no return value is needed. Except for onchange methods that can optionally return a dict with a warning message to display in the user interface. As an example, we can use this to perform some automation in the To-Do form: when Responsible is set to an empty value, we will also empty the team list. For this, edit the todo_stage/models/todo_task_model.py file to add the following method: @api.onchange('user_id') def onchange_user_id(self): if not user_id: self.team_ids = None return { 'warning': { 'title': 'No Responsible', 'message': 'Team was also reset.' } } Here, we are using the @api.onchange decorator to attach some logic to any changes in the user_id field, when done through the user interface. Note that the actual method name is not relevant, but the convention is for its name to begin with onchange_. Inside an onchange method, self represents a single virtual record containing all the fields currently set in the record being edited, and we can interact with them. Most of the time, this is what we want to do: to automatically fill values in other fields, depending on the value set to the changed field. In this case, we are setting the team_ids field to an empty value. The onchange methods don't need to return anything, but they can return a dictionary containing a warning or a domain key: The warning key should describe a message to show in a dialogue window, such as: {'title': 'Message Title', 'message': 'Message Body'}. The domain key can set or change the domain attribute of other fields. This allows you to build more user-friendly interfaces, by having to-many fields list only the selection option that make sense for this case. The value for the domain key looks like this: {'team_ids': [('is_author', '=', True)]} Using the ORM built-in methods The decorators discussed in the previous section allow us to add certain features to our models, such as implementing validations and automatic computations. We also have the basic methods provided by the ORM, used mainly to perform CRUD (create, read, update and delete) operations on our model data. To read data, the main methods provided are search() and browse(). Now we will explore the write operations provided by the ORM, and how they can be extended to support custom logic. Methods for writing model data The ORM provides three methods for the three basic write operations: <Model>.create(values) creates a new record on the model. Returns the created record. <Recordset>.write(values) updates field values on the recordset. Returns nothing. <Recordset>.unlink() deletes the records from the database. Returns nothing. The values argument is a dictionary, mapping field names to values to write. In some cases, we need to extend these methods to add some business logic to be triggered whenever these actions are executed. By placing our logic in the appropriate section of the custom method, we can have the code run before or after the main operations are executed. Using the TodoTask model as an example, we can make a custom create(), which would look like this: @api.model def create(self, vals): # Code before create: should use the `vals` dict new_record = super(TodoTask, self).create(vals) # Code after create: can use the `new_record` created return new_record Python 3 introduced a simplified way to use super() that could have been used in the preceding code samples. We chose to use the Python 2 compatible form. If we don't mind breaking Python 2 support for our code, we can use the simplified form, without the arguments referencing the class name and self. For example: super().create(vals) A custom write() would follow this structure: @api.multi def write(self, vals): # Code before write: can use `self`, with the old values super(TodoTask, self).write(vals) # Code after write: can use `self`, with the updated values return True While extending create() and write()  opens up a lot of possibilities, remember in many cases we don't need to do that, since there are tools also available that may be better suited: For field values that are automatically calculated based on other fields, we should use computed fields. An example of this is to calculate a header total when the values of the lines are changed. To have field default values calculated dynamically, we can use a field default bound to a function instead of a fixed value. To have values set on other fields when a field is changed, we can use onchange functions. An example of this is when picking a customer, setting their currency as the document's currency that can later be manually changed by the user. Keep in mind that on change only works on form view interaction and not on direct write calls. For validations, we should use constraint functions decorated with @api.constraints(fld1,fld2,...). These are like computed fields but, instead of computing values, they are expected to raise errors. Consider carefully if you really need to use extensions to the create or write methods. In most cases, we just need to perform some validation or automatically compute some value, when the record is saved. But we have better tools for this: validations are best implemented with @api.constrains methods, and automatic calculations are better implemented as computed fields. In this case, we need to compute field values when saving. If, for some reason, computed fields are not a valid solution, the best approach is to have our logic at the top of the method, accumulating the changes needed into the vals dictionary that will be passed to the final super() call. For the write() method, having further write operations on the same model will lead to a recursion loop and end with an error when the worker process resources are exhausted. Please consider if this is really needed. If it is, a technique to avoid the recursion loop is to set a flag in the context. For example, we could add code such as the following: if not self.env.context.get('todo_task_writing'): self.with_context(todo_task_writing=True).write( some_values) With this technique, our specific logic is guarded by an if statement, and runs only if a specific marker is not found in the context. Furthermore, our self.write() operations should use with_context to set that marker. This combination ensures that the custom login inside the if statement runs only once, and is not triggered on further write() calls, avoiding the infinite loop. These are common extension examples, but of course any standard method available for a model can be inherited in a similar way to add our custom logic to it. Methods for web client use over RPC We have seen the most important model methods used to generate recordsets and how to write to them, but there are a few more model methods available for more specific actions, as shown here: read([fields]) is similar to the browse method, but, instead of a recordset, it returns a list of rows of data with the fields given as its argument. Each row is a dictionary. It provides a serialized representation of the data that can be sent through RPC protocols and is intended to be used by client programs and not in server logic. search_read([domain], [fields], offset=0, limit=None, order=None) performs a search operation followed by a read on the resulting record list. It is intended to be used by RPC clients and saves them the extra round trip needed when doing a search followed by a read on the results. Methods for data import and export The import and export operations, are also available from the ORM API, through the following methods: load([fields], [data]) is used to import data acquired from a CSV file. The first argument is the list of fields to import, and it maps directly to a CSV top row. The second argument is a list of records, where each record is a list of string values to parse and import, and it maps directly to the CSV data rows and columns. It implements the features of CSV data import, such as the external identifiers support. It is used by the web client Import feature. export_data([fields], raw_data=False) is used by the web client Export function. It returns a dictionary with a data key containing the data: a list of rows. The field names can use the .id and /id suffixes used in CSV files, and the data is in a format compatible with an importable CSV file. The optional raw_data argument allows for data values to be exported with their Python types, instead of the string representation used in CSV. Methods for the user interface The following methods are mostly used by the web client to render the user interface and perform basic interaction: name_get()  returns a list of (ID, name) tuples with the text representing each record. It is used by default for computing the display_name value, providing the text representation of relation fields. It can be extended to implement custom display representations, such as displaying the record code and name instead of only the name. name_search(name='', args=None, operator='ilike', limit=100) returns a list of (ID, name) tuples, where the display name matches the text in the name argument. It is used in the UI while typing in a relation field to produce the list with the suggested records matching the typed text. For example, it is used to implement product lookup both by name and by reference, while typing in a field to pick a product. name_create(name) creates a new record with only the title name to use for it. It is used in the UI for the "quick-create" feature, where you can quickly create a related record by just providing its name. It can be extended to provide specific defaults for the new records created through this feature. default_get([fields]) returns a dictionary with the default values for a new record to be created. The default values may depend on variables such as the current user or the session context. fields_get() is used to describe the model's field definitions, as seen in the View Fields option of the developer menu. fields_view_get() is used by the web client to retrieve the structure of the UI view to render. It can be given the ID of the view as an argument or the type of view we want using view_type='form'. For example, you may try this: self.fields_view_get(view_type='tree'). The Mail and Social features API Odoo has available global messaging and activity planning features, provided by the Discuss app, with the technical name mail. The mail module provides the mail.thread abstract class that makes it simple to add the messaging features to any model. To add the mail.thread features to the To-Do tasks, we just need to inherit from it: class TodoTask(models.Model): _name = 'todo.task' _inherit = ['todo.task', 'mail.thread'] After this, among other things, our model will have two new fields available. For each record (sometimes also called a document) we have: mail_follower_ids stores the followers, and corresponding notification preferences mail_message_ids lists all the related messages The followers can be either partners or channels. A partner represents a specific person or organization. A channel is not a particular person, and instead represents a subscription list. Each follower also has a list of message types that they are subscribed to. Only the selected message types will generate notifications for them. Message subtypes Some types of messages are called subtypes. They are stored in the mail.message.subtype model and accessible in the Technical | Email | Subtypes menu. By default, we have three message subtypes available: Discussions, with mail.mt_comment XMLID, used for the messages created with the Send message link. It is intended to send a notification. Activities, with mail.mt_activities XMLID, used for the messages created with the Schedule activity link. It is intended to send a notification. Note, with mail.mt_note XMLID, used for the messages created with the Log note link. It is not intended to send a notification. Subtypes have the default notification settings described previously, but users are able to change them for specific documents, for example, to mute a discussion they are not interested in. Other than the built-in subtypes, we can also add our own subtypes to customize the notifications for our apps. Subtypes can be generic or intended for a particular model. For the latter case, we should fill in the subtype's res_model field with the name of the model it should apply to. Posting messages Our business logic can make use of this messaging system to send notifications to users. To post a message we use the message_post() method. For example: self.message_post('Hello!') This adds a simple text message, but sends no notification to the followers. That is because by default the mail.mt_note subtype is used for the posted messages. But we can have the message posted with the particular subtype we want. To add a message and have it send notifications to the followers, we should use the following: self.message_post('Hello again!', subtype='mail.mt_comment') We can also add a subject line to the message by adding the subject parameter. The message body is HTML, so we can include markup for text effects, such as <b> for bold text or <i> for italics. The message body will be sanitized for security reasons, so some particular HTML elements may not make it to the final message. Adding followers Also interesting from a business logic viewpoint is the ability to automatically add followers to a document, so that they can then get the corresponding notifications. For this we have several methods available to add followers: message_subscribe(partner_ids=<list of int IDs>) adds Partners message_subscribe(channel_ids=<list of int IDs>) adds Channels message_subscribe_users(user_ids=<list of int IDs>) adds Users The default subtypes will be used. To force subscribing a specific list of subtypes, just add the subtype_ids=<list of int IDs> with the specific subtypes you want to be subscribed. In this article, we went through an explanation of the features the ORM API proposes, and how they can be used when creating our models. We also learned about the mail module and the global messaging features it provides. To look further into ORM, and have a deeper understanding of how recordsets work and can be manipulated, read our book Odoo 11 Development Essentials - Third Edition. ERP tool in focus: Odoo 11 Building Your First Odoo Application How to Scaffold a New module in Odoo 11
Read more
  • 0
  • 0
  • 36847

article-image-introduction-nodejs-design-patterns
Packt
18 Feb 2016
27 min read
Save for later

An Introduction to Node.js Design Patterns

Packt
18 Feb 2016
27 min read
A design pattern is a reusable solution to a recurring problem; the term is really broad in its definition and can span multiple domains of application. However, the term is often associated with a well-known set of object-oriented patterns that were popularized in the 90's by the book, Design Patterns: Elements of Reusable Object-Oriented Software, Pearson Education by the almost legendary Gang of Four (GoF): Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. We will often refer to these specific set of patterns as traditional design patterns, or GoF design patterns. (For more resources related to this topic, see here.) Applying this set of object-oriented design patterns in JavaScript is not as linear and formal as it would happen in a class-based object-oriented language. As we know, JavaScript is multi-paradigm, object-oriented, and prototype-based, and has dynamic typing; it treats functions as first class citizens, and allows functional programming styles. These characteristics make JavaScript a very versatile language, which gives tremendous power to the developer, but at the same time, it causes a fragmentation of programming styles, conventions, techniques, and ultimately the patterns of its ecosystem. There are so many ways to achieve the same result using JavaScript that everybody has their own opinion on what the best way is to approach a problem. A clear demonstration of this phenomenon is the abundance of frameworks and opinionated libraries in the JavaScript ecosystem; probably, no other language has ever seen so many, especially now that Node.js has given new astonishing possibilities to JavaScript and has created so many new scenarios. In this context, the traditional design patterns too are affected by the nature of JavaScript. There are so many ways in which they can be implemented so that their traditional, strongly object-oriented implementation is not a pattern anymore, and in some cases, not even possible because JavaScript, as we know, doesn't have real classes or abstract interfaces. What doesn't change though, is the original idea at the base of each pattern, the problem it solves, and the concepts at the heart of the solution. In this article, we will see how some of the most important GoF design patterns apply to Node.js and its philosophy, thus rediscovering their importance from another perspective. The design patterns explored in this article are as follows: Factory Proxy Decorator Adapter Strategy State Template Middleware Command This article assumes that the reader has some notion of how inheritance works in JavaScript. Please also be advised that throughout this article we will often use generic and more intuitive diagrams to describe a pattern in place of standard UML, since many patterns can have an implementation based not only on classes, but also on objects and even functions. Factory We begin our journey starting from what probably is the most simple and common design pattern in Node.js: Factory. A generic interface for creating objects We already stressed the fact that, in JavaScript, the functional paradigm is often preferred to a purely object-oriented design, for its simplicity, usability, and small surface area. This is especially true when creating new object instances. In fact, invoking a factory, instead of directly creating a new object from a prototype using the new operator or Object.create(), is so much more convenient and flexible under several aspects. First and foremost, a factory allows us to separate the object creation from its implementation; essentially, a factory wraps the creation of a new instance, giving us more flexibility and control in the way we do it. Inside the factory, we can create a new instance leveraging closures, using a prototype and the new operator, using Object.create(), or even returning a different instance based on a particular condition. The consumer of the factory is totally agnostic about how the creation of the instance is carried out. The truth is that, by using new, we are binding our code to one specific way of creating an object, while in JavaScript we can have much more flexibility, almost for free. As a quick example, let's consider a simple factory that creates an Image object: function createImage(name) { return new Image(name); } var image = createImage('photo.jpeg'); The createImage() factory might look totally unnecessary, why not instantiate the Image class by using the new operator directly? Something like the following line of code: var image = new Image(name); As we already mentioned, using new binds our code to one particular type of object; in the preceding case, to objects of type, Image. A factory instead, gives us much more flexibility; imagine that we want to refactor the Image class, splitting it into smaller classes, one for each image format that we support. If we exposed a factory as the only means to create new images, we can simply rewrite it as follows, without breaking any of the existing code: function createImage(name) { if(name.match(/.jpeg$/)) { return new JpegImage(name); } else if(name.match(/.gif$/)) { return new GifImage(name); } else if(name.match(/.png$/)) { return new PngImage(name); } else { throw new Exception('Unsupported format'); } } Our factory also allows us to not expose the constructors of the objects it creates, and prevents them from being extended or modified (remember the principle of small surface area?). In Node.js, this can be achieved by exporting only the factory, while keeping each constructor private. A mechanism to enforce encapsulation A factory can also be used as an encapsulation mechanism, thanks to closures. Encapsulation refers to the technique of controlling the access to some internal details of an object by preventing the external code from manipulating them directly. The interaction with the object happens only through its public interface, isolating the external code from the changes in the implementation details of the object. This practice is also referred to as information hiding. Encapsulation is also a fundamental principle of object-oriented design, together with inheritance, polymorphism, and abstraction. As we know, in JavaScript, we don't have access level modifiers (for example, we can't declare a private variable), so the only way to enforce encapsulation is through function scopes and closures. A factory makes it straightforward to enforce private variables, consider the following code for example: function createPerson(name) { var privateProperties = {}; var person = { setName: function(name) { if(!name) throw new Error('A person must have a name'); privateProperties.name = name; }, getName: function() { return privateProperties.name; } }; person.setName(name); return person; } In the preceding code, we leverage closures to create two objects: a person object which represents the public interface returned by the factory and a group of privateProperties that are inaccessible from the outside and that can be manipulated only through the interface provided by the person object. For example, in the preceding code, we make sure that a person's name is never empty, this would not be possible to enforce if name was just a property of the person object. Factories are only one of the techniques that we have for creating private members; in fact, other possible approaches are as follows: Defining private variables in a constructor (as recommended by Douglas Crockford: http://javascript.crockford.com/private.html) Using conventions, for example, prefixing the name of a property with an underscore "_" or the dollar sign "$" (this however, does not technically prevent a member from being accessed from the outside) Using ES6 WeakMaps (http://fitzgeraldnick.com/weblog/53/) Building a simple code profiler Now, let's work on a complete example using a factory. Let's build a simple code profiler, an object with the following properties: A start() method that triggers the start of a profiling session An end() method to terminate the session and log its execution time to the console Let's start by creating a file named profiler.js, which will have the following content: function Profiler(label) { this.label = label; this.lastTime = null; } Profiler.prototype.start = function() { this.lastTime = process.hrtime(); } Profiler.prototype.end = function() { var diff = process.hrtime(this.lastTime); console.log('Timer "' + this.label + '" took ' + diff[0] + ' seconds and ' + diff[1] + ' nanoseconds.'); } There is nothing fancy in the preceding class; we simply use the default high resolution timer to save the current time when start() is invoked, and then calculate the elapsed time when end() is executed, printing the result to the console. Now, if we are going to use such a profiler in a real-world application to calculate the execution time of the different routines, we can easily imagine the huge amount of logging we will generate to the standard output, especially in a production environment. What we might want to do instead is redirect the profiling information to another source, for example, a database, or alternatively, disabling the profiler altogether if the application is running in the production mode. It's clear that if we were to instantiate a Profiler object directly by using the new operator, we would need some extra logic in the client code or in the Profiler object itself in order to switch between the different logics. We can instead use a factory to abstract the creation of the Profiler object, so that, depending on whether the application runs in the production or development mode, we can return a fully working Profiler object, or alternatively, a mock object with the same interface, but with empty methods. Let's do this then, in the profiler.js module instead of exporting the Profiler constructor, we will export only a function, our factory. The following is its code: module.exports = function(label) { if(process.env.NODE_ENV === 'development') { return new Profiler(label); //[1] } else if(process.env.NODE_ENV === 'production') { return { //[2] start: function() {}, end: function() {} } } else { throw new Error('Must set NODE_ENV'); } } The factory that we created abstracts the creation of a profiler object from its implementation: If the application is running in the development mode, we return a new, fully functional Profiler object If instead the application is running in the production mode, we return a mock object where the start() and stop() methods are empty functions The nice feature to highlight is that, thanks to the JavaScript dynamic typing, we were able to return an object instantiated with the new operator in one circumstance and a simple object literal in the other (this is also known as duck typing http://en.wikipedia.org/wiki/Duck_typing). Our factory is doing its job perfectly; we can really create objects in any way that we like inside the factory function, and we can execute additional initialization steps or return a different type of object based on particular conditions, and all of this while isolating the consumer of the object from all these details. We can easily understand the power of this simple pattern. Now, we can play with our profiler; this is a possible use case for the factory that we just created earlier: var profiler = require('./profiler'); function getRandomArray(len) { var p = profiler('Generating a ' + len + ' items long array'); p.start(); var arr = []; for(var i = 0; i < len; i++) { arr.push(Math.random()); } p.end(); } getRandomArray(1e6); console.log('Done'); The p variable contains the instance of our Profiler object, but we don't know how it's created and what its implementation is at this point in the code. If we include the preceding code in a file named profilerTest.js, we can easily test these assumptions. To try the program with profiling enabled, run the following command: export NODE_ENV=development; node profilerTest The preceding command enables the real profiler and prints the profiling information to the console. If we want to try the mock profiler instead, we can run the following command: export NODE_ENV=production; node profilerTest The example that we just presented is just a simple application of the factory function pattern, but it clearly shows the advantages of separating an object's creation from its implementation. In the wild As we said, factories are very popular in Node.js. Many packages offer only a factory for creating new instances, some examples are the following: Dnode (https://npmjs.org/package/dnode): This is an RPC system for Node.js. If we look into its source code, we will see that its logic is implemented into a class named D; however, this is never exposed to the outside as the only exported interface is a factory, which allows us to create new instances of the class. You can take a look at its source code at https://github.com/substack/dnode/blob/34d1c9aa9696f13bdf8fb99d9d039367ad873f90/index.js#L7-9. Restify (https://npmjs.org/package/restify): This is a framework to build REST API that allows us to create new instances of a server using the restify.createServer()factory, which internally creates a new instance of the Server class (which is not exported). You can take a look at its source code at https://github.com/mcavage/node-restify/blob/5f31e2334b38361ac7ac1a5e5d852b7206ef7d94/lib/index.js#L91-116. Other modules expose both a class and a factory, but document the factory as the main method—or the most convenient way—to create new instances; some of the examples are as follows: http-proxy (https://npmjs.org/package/http-proxy): This is a programmable proxying library, where new instances are created with httpProxy.createProxyServer(options). The core Node.js HTTP server: This is where new instances are mostly created using http.createServer(), even though this is essentially a shortcut for new http.Server(). bunyan (https://npmjs.org/package/bunyan): This is a popular logging library; in its readme file the contributors propose a factory, bunyan.createLogger(), as the main method to create new instances, even though this would be equivalent to running new bunyan(). Some other modules provide a factory to wrap the creation of other components. Popular examples are through2 and from2 , which allow us to simplify the creation of new streams using a factory approach, freeing the developer from explicitly using inheritance and the new operator. Proxy A proxy is an object that controls the access to another object called subject. The proxy and the subject have an identical interface and this allows us to transparently swap one for the other; in fact, the alternative name for this pattern is surrogate. A proxy intercepts all or some of the operations that are meant to be executed on the subject, augmenting or complementing their behavior. The following figure shows the diagrammatic representation: The preceding figure shows us how the Proxy and the Subject have the same interface and how this is totally transparent to the client, who can use one or the other interchangeably. The Proxy forwards each operation to the subject, enhancing its behavior with additional preprocessing or post-processing. It's important to observe that we are not talking about proxying between classes; the Proxy pattern involves wrapping actual instances of the subject, thus preserving its state. A proxy is useful in several circumstances, for example, consider the following ones: Data validation: The proxy validates the input before forwarding it to the subject Security: The proxy verifies that the client is authorized to perform the operation and it passes the request to the subject only if the outcome of the check is positive Caching: The proxy keeps an internal cache so that the operations are executed on the subject only if the data is not yet present in the cache Lazy initialization: If the creation of the subject is expensive, the proxy can delay it to when it's really necessary Logging: The proxy intercepts the method invocations and the relative parameters, recoding them as they happen Remote objects: A proxy can take an object that is located remotely, and make it appear local Of course, there are many more applications for the Proxy pattern, but these should give us an idea of the extent of its purpose. Techniques for implementing proxies When proxying an object, we can decide to intercept all its methods or only part of them, while delegating the rest of them directly to the subject. There are several ways in which this can be achieved; let's analyze some of them. Object composition Composition is the technique whereby an object is combined with another object for the purpose of extending or using its functionality. In the specific case of the Proxy pattern, a new object with the same interface as the subject is created, and a reference to the subject is stored internally in the proxy in the form of an instance variable or a closure variable. The subject can be injected from the client at creation time or created by the proxy itself. The following is one example of this technique using a pseudo class and a factory: function createProxy(subject) { var proto = Object.getPrototypeOf(subject); function Proxy(subject) { this.subject = subject; } Proxy.prototype = Object.create(proto); //proxied method Proxy.prototype.hello = function() { return this.subject.hello() + ' world!'; } //delegated method Proxy.prototype.goodbye = function() { return this.subject.goodbye .apply(this.subject, arguments); } return new Proxy(subject); } To implement a proxy using composition, we have to intercept the methods that we are interested in manipulating (such as hello()), while simply delegating the rest of them to the subject (as we did with goodbye()). The preceding code also shows the particular case where the subject has a prototype and we want to maintain the correct prototype chain, so that, executing proxy instanceof Subject will return true; we used pseudo-classical inheritance to achieve this. This is just an extra step, required only if we are interested in maintaining the prototype chain, which can be useful in order to improve the compatibility of the proxy with code initially meant to work with the subject. However, as JavaScript has dynamic typing, most of the time we can avoid using inheritance and use more immediate approaches. For example, an alternative implementation of the proxy presented in the preceding code, might just use an object literal and a factory: function createProxy(subject) { return { //proxied method hello: function() { return subject.hello() + ' world!'; }, //delegated method goodbye: function() { return subject.goodbye.apply(subject, arguments); } }; } If we want to create a proxy that delegates most of its methods, it would be convenient to generate these automatically using a library, such as delegates (https://npmjs.org/package/delegates). Object augmentation Object augmentation (or monkey patching)is probably the most pragmatic way of proxying individual methods of an object and consists of modifying the subject directly by replacing a method with its proxied implementation; consider the following example: function createProxy(subject) { var helloOrig = subject.hello; subject.hello = function() { return helloOrig.call(this) + ' world!'; } return subject; } This technique is definitely the most convenient one when we need to proxy only one or a few methods, but it has the drawback of modifying the subject object directly. A comparison of the different techniques Composition can be considered the safest way of creating a proxy, because it leaves the subject untouched without mutating its original behavior. Its only drawback is that we have to manually delegate all the methods, even if we want to proxy only one of them. If needed, we might also have to delegate the access to the properties of the subject. The object properties can be delegated using Object.defineProperty(). Find out more at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/defineProperty. Object augmentation, on the other hand, modifies the subject, which might not always be what we want, but it does not present the various inconveniences related to delegation. For this reason, object augmentation is definitely the most pragmatic way to implement proxies in JavaScript, and it's the preferred technique in all those circumstances where modifying the subject is not a big concern. However, there is at least one situation where composition is almost necessary; this is when we want to control the initialization of the subject as for example, to create it only when needed (lazy initialization). It is worth pointing out that by using a factory function (createProxy() in our examples), we can shield our code from the technique used to generate the proxy. Creating a logging Writable stream To see the proxy pattern in a real example, we will now build an object that acts as a proxy to a Writable stream, by intercepting all the calls to the write() method and logging a message every time this happens. We will use an object composition to implement our proxy; this is how the loggingWritable.js file looks: function createLoggingWritable(writableOrig) { var proto = Object.getPrototypeOf(writableOrig); function LoggingWritable(subject) { this.writableOrig = writableOrig; } LoggingWritable.prototype = Object.create(proto); LoggingWritable.prototype.write = function(chunk, encoding, callback) { if(!callback && typeof encoding === 'function') { callback = encoding; encoding = undefined; } console.log('Writing ', chunk); return this.writableOrig.write(chunk, encoding, function() { console.log('Finished writing ', chunk); callback && callback(); }); }; LoggingWritable.prototype.on = function() { return this.writableOrig.on .apply(this.writableOrig, arguments); }; LoggingWritable.prototype.end = function() { return this.writableOrig.end .apply(this.writableOrig, arguments); } return new LoggingWritable(this.writableOrig); } In the preceding code, we created a factory that returns a proxied version of the writable object passed as an argument. We provide an override for the write() method that logs a message to the standard output every time it is invoked and every time the asynchronous operation completes. This is also a good example to demonstrate the particular case of creating proxies of asynchronous functions, which makes necessary to proxy the callback as well; this is an important detail to be considered in a platform such as Node.js. The remaining methods, on() and end(), are simply delegated to the original writable stream (to keep the code leaner we are not considering the other methods of the Writable interface). We can now include a few more lines of code into the logginWritable.js module to test the proxy that we just created: var fs = require('fs'); var writable = fs.createWriteStream('test.txt'); var writableProxy = createProxy(writable); writableProxy.write('First chunk'); writableProxy.write('Second chunk'); writable.write('This is not logged'); writableProxy.end(); The proxy did not change the original interface of the stream or its external behavior, but if we run the preceding code, we will now see that every chunk that is written into the stream is transparently logged to the console. Proxy in the ecosystem – function hooks and AOP In its numerous forms, Proxy is a quite popular pattern in Node.js and in the ecosystem. In fact, we can find several libraries that allow us to simplify the creation of proxies, most of the time leveraging object augmentation as an implementation approach. In the community, this pattern can be also referred to as function hooking or sometimes also as Aspect Oriented Programming (AOP), which is actually a common area of application for proxies. As it happens in AOP, these libraries usually allow the developer to set pre or post execution hooks for a specific method (or a set of methods) that allow us to execute some custom code before and after the execution of the advised method respectively. Sometimes proxies are also called middleware, because, as it happens in the middleware pattern, they allow us to preprocess and post-process the input/output of a function. Sometimes, they also allow to register multiple hooks for the same method using a middleware-like pipeline. There are several libraries on npm that allow us to implement function hooks with little effort. Among them there are hooks (https://npmjs.org/package/hooks), hooker (https://npmjs.org/package/hooker), and meld (https://npmjs.org/package/meld). In the wild Mongoose (http://mongoosejs.com) is a popular Object-Document Mapping (ODM) library for MongoDB. Internally, it uses the hooks package (https://npmjs.org/package/hooks) to provide pre and post execution hooks for the init, validate, save, and remove methods of its Document objects. Find out more on the official documentation at http://mongoosejs.com/docs/middleware.html. Decorator Decorator is a structural pattern that consists of dynamically augmenting the behavior of an existing object. It's different from classical inheritance, because the behavior is not added to all the objects of the same class but only to the instances that are explicitly decorated. Implementation-wise, it is very similar to the Proxy pattern, but instead of enhancing or modifying the behavior of the existing interface of an object, it augments it with new functionalities, as described in the following figure: In the previous figure, the Decorator object is extending the Component object by adding the methodC() operation. The existing methods are usually delegated to the decorated object, without further processing. Of course, if necessary we can easily combine the Proxy pattern, so that also the calls to the existing methods can be intercepted and manipulated. Techniques for implementing decorators Although Proxy and Decorator are conceptually two different patterns, with different intents, they practically share the same implementation strategies. Let's revise them. Composition Using composition, the decorated component is wrapped around a new object that usually inherits from it. The Decorator in this case simply needs to define the new methods while delegating the existing ones to the original component: function decorate(component) { var proto = Object.getPrototypeOf(component); function Decorator(component) { this.component = component; } Decorator.prototype = Object.create(proto); //new method Decorator.prototype.greetings = function() { //... }; //delegated method Decorator.prototype.hello = function() { this.component.hello.apply(this.component, arguments); }; return new Decorator(component); } Object augmentation Object decoration can also be achieved by simply attaching new methods directly to the decorated object, as follows: function decorate(component) { //new method component.greetings = function() { //... }; return component; } The same caveats discussed during the analysis of the Proxy pattern are also valid for Decorator. Let's now practice the pattern with a working example! Decorating a LevelUP database Before we start coding with the next example, let's spend a few words to introduce LevelUP, the module that we are now going to work with. Introducing LevelUP and LevelDB LevelUP (https://npmjs.org/package/levelup) is a Node.js wrapper around Google's LevelDB, a key-value store originally built to implement IndexedDB in the Chrome browser, but it's much more than that. LevelDB has been defined by Dominic Tarr as the "Node.js of databases", because of its minimalism and extensibility. Like Node.js, LevelDB provides blazing fast performances and only the most basic set of features, allowing developers to build any kind of database on top of it. The Node.js community, and in this case Rod Vagg, did not miss the chance to bring the power of this database into Node.js by creating LevelUP. Born as a wrapper for LevelDB, it then evolved to support several kinds of backends, from in-memory stores, to other NoSQL databases such as Riak and Redis, to web storage engines such as IndexedDB and localStorage, allowing to use the same API on both the server and the client, opening up some really interesting scenarios. Today, there is a full-fledged ecosystem around LevelUp made of plugins and modules that extend the tiny core to implement features such as replication, secondary indexes, live updates, query engines, and more. Also, complete databases were built on top of LevelUP, including CouchDB clones such as PouchDB (https://npmjs.org/package/pouchdb) and CouchUP (https://npmjs.org/package/couchup), and even a graph database, levelgraph (https://npmjs.org/package/levelgraph) that can work both on Node.js and the browser! Find out more about the LevelUP ecosystem at https://github.com/rvagg/node-levelup/wiki/Modules. Implementing a LevelUP plugin In the next example, we are going to show how we can create a simple plugin for LevelUp using the Decorator pattern, and in particular, the object augmentation technique, which is the simplest but nonetheless the most pragmatic and effective way to decorate objects with additional capabilities. For convenience, we are going to use the level package (http://npmjs.org/package/level) that bundles both levelup and the default adapter called leveldown, which uses LevelDB as the backend. What we want to build is a plugin for LevelUP that allows to receive notifications every time an object with a certain pattern is saved into the database. For example, if we subscribe to a pattern such as {a: 1}, we want to receive a notification when objects such as {a: 1, b: 3} or {a: 1, c: 'x'} are saved into the database. Let's start to build our small plugin by creating a new module called levelSubscribe.js. We will then insert the following code: module.exports = function levelSubscribe(db) { db.subscribe = function(pattern, listener) { //[1] db.on('put', function(key, val) { //[2] var match = Object.keys(pattern).every(function(k) { //[3] return pattern[k] === val[k]; }); if(match) { listener(key, val); //[4] } }); }; return db; } That's it for our plugin, and it's extremely simple. Let's see what happens in the preceding code briefly: We decorated the db object with a new method named subscribe(). We simply attached the method directly to the provided db instance (object augmentation). We listen for any put operation performed on the database. We performed a very simple pattern-matching algorithm, which verified that all the properties in the provided pattern are also available on the data being inserted. If we have a match, we notify the listener. Let's now create some code—in a new file named levelSubscribeTest.js—to try out our new plugin: var level = require('level'); //[1] var db = level(__dirname + '/db', {valueEncoding: 'json'}); var levelSubscribe = require('./levelSubscribe'); //[2] db = levelSubscribe(db); db.subscribe({doctype: 'tweet', language: 'en'}, //[3] function(k, val){ console.log(val); }); //[4] db.put('1', {doctype: 'tweet', text: 'Hi', language: 'en'}); db.put('2', {doctype: 'company', name: 'ACME Co.'}); This is what we did in the preceding code: First, we initialize our LevelUP database, choosing the directory where the files will be stored and the default encoding for the values. Then, we attach our plugin, which decorates the original db object. At this point, we are ready to use the new feature provided by our plugin, the subscribe() method, where we specify that we are interested in all the objects with doctype: 'tweet' and language: 'en'. Finally, we save some values in the database, so that we can see our plugin in action: db.put('1', {doctype: 'tweet', text: 'Hi', language: 'en'}); db.put('2', {doctype: 'company', name: 'ACME Co.'}); This example shows a real application of the decorator pattern in its most simple implementation: object augmentation. It might look like a trivial pattern but it has undoubted power if used appropriately. For simplicity, our plugin will work only in combination with the put operations, but it can be easily expanded to work even with the batch operations (https://github.com/rvagg/node-levelup#batch). In the wild For more examples of how Decorator is used in the real world, we might want to inspect the code of some more LevelUp plugins: level-inverted-index (https://github.com/dominictarr/level-inverted-index): This is a plugin that adds inverted indexes to a LevelUP database, allowing to perform simple text searches across the values stored in the database level-plus (https://github.com/eugeneware/levelplus): This is a plugin that adds atomic updates to a LevelUP database Summary To learn more about Node.js, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Node.js Essentials (https://www.packtpub.com/web-development/nodejs-essentials) Node.js Blueprints (https://www.packtpub.com/web-development/nodejs-blueprints) Learning Node.js for Mobile Application Development (https://www.packtpub.com/web-development/learning-nodejs-mobile-application-development) Mastering Node.js (https://www.packtpub.com/web-development/mastering-nodejs) Resources for Article: Further resources on this subject: Node.js Fundamentals [Article] Learning Node.js for Mobile Application Development [Article] Developing a Basic Site with Node.js and Express [Article]
Read more
  • 0
  • 0
  • 36289

article-image-developers-from-the-swift-for-tensorflow-project-propose-adding-first-class-differentiable-programming-to-swift
Bhagyashree R
09 Sep 2019
5 min read
Save for later

Developers from the Swift for TensorFlow project propose adding first-class differentiable programming to Swift

Bhagyashree R
09 Sep 2019
5 min read
After working for over 1.5 years on the Differentiable Programming Mega-Proposal, Richard Wei, a developer at Google Brain, and his team submitted the proposal on the Swift Evolution forum on Thursday last week. This proposal aims to “push Swift's capabilities to the next level in numerics and machine learning” by introducing differentiable programming as a new language feature in Swift. It is a part of the Swift for TensorFlow project under which the team is integrating TensorFlow directly into the language to offer developers a next-generation platform for machine learning. What is differentiable programming With the increasing sophistication in deep learning models and the introduction of modern deep learning frameworks, many researchers have started to realize that building neural networks is very similar to programming. Yann LeCun, VP and Chief AI Scientist at Facebook, calls differentiable programming “a little more than a rebranding of the modern collection Deep Learning techniques, the same way Deep Learning was a rebranding of the modern incarnations of neural nets with more than two layers.” He compares it with regular programming, with the only difference that the resulting programs are “parameterized, automatically differentiated, and trainable/optimizable.” Many also say that differentiable programming is a different name for automatic differentiation, a collection of techniques to numerically evaluate the derivative of a function. It can be seen as a new programming paradigm in which programs can be differentiated throughout. Check out the paper “Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagation” to get a better understanding of differentiable programming. Why differential programming is proposed in Swift Swift is an expressive, high-performance language, which makes it a perfect candidate for numerical applications. According to the proposal authors, first-class support for differentiable programming in Swift will allow safe and powerful machine learning development. The authors also believe that this is a “big step towards high-level numerical computing support.” With this proposal, they aim to make Swift a “real contender in the numerical computing and machine learning landscape.” Here are some of the advantages of adding first-class support for differentiable programming in Swift: Better language coverage: First-class differentiable programming support will enable differentiation to work smoothly with other Swift features. This will allow developers to code normally without being restricted to a subset of Swift. Enable extensibility: This will provide developers an “extensible differentiable programming system.” They will be able to create custom differentiation APIs by leveraging primitive operators defined in the standard library and supported by the type system. Static warnings and errors: This will enable the compiler to statically identify the functions that cannot be differentiated or will give a zero derivative. It will then be able to give a non-differentiability error or warning. This will improve productivity by making common runtime errors in machine learning directly debuggable without library boundaries. Some of the components that will be added in Swift under this proposal are: The Differentiable protocol: This is a standard library protocol that will generalize all data structures that can be a parameter or result of a differentiable function. The @differentiable declaration attribute: This will be used to mark all the function-like declarations as differentiable. The @differentiable function types: This is a subtype of normal function types with a different runtime representation and calling convention. Differentiable function types will have differentiable parameters and results. Differential operators: These are the core differentiation APIs that take ‘@differentiable’ functions as inputs and return derivative functions or compute derivative values. @differentiating and @transposing attributes: These attributes are for declaring custom derivative function for some other function declaration. This proposal sparked a discussion on Hacker News. Many developers were excited about bringing differentiable programming support in the Swift core. A user commented, “This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully, greatly improve ML APIs while they're at it.” Some felt that a library could have served the purpose. “I don't see why a well-written library could not serve the same purpose. It seems like a lot of cruft. I doubt, for example, Python would ever consider adding this and it's the de facto language that would benefit the most from something like this - due to the existing tools and communities. It just seems so narrow and not at the same level of abstraction that languages typically sit at. I could see the language supporting higher-level functionality so a library could do this without a bunch of extra work (such as by some reflection),” a user added. Users also discussed another effort that goes along the lines of this project: Julia Zygote, which is a working prototype for source-to-source automatic differentiation. A user commented, “Yup, work is continuing apace with Julia’s next-gen Zygote project. Also, from the GP’s thought about applications beyond DL, my favorite examples so far are for model-based RL and Neural ODEs.” To know more in detail, check out the proposal: Differentiable Programming Mega-Proposal. Other news in programming Why Perl 6 is considering a name change? The Julia team shares its finalized release process with the community TypeScript 3.6 releases with stricter generators, new functions in TypeScript playground, better Unicode support for identifiers, and more
Read more
  • 0
  • 0
  • 34945

article-image-factory-method-pattern
Packt
10 Feb 2016
10 min read
Save for later

The Factory Method Pattern

Packt
10 Feb 2016
10 min read
In this article by Anshul Verma and Jitendra Zaa, author of the book Apex Design Patterns, we will discuss some problems that can occur mainly during the creation of class instances and how we can write the code for the creation of objects in a more simple, easy to maintain, and scalable way. (For more resources related to this topic, see here.) In this article, we will discuss the the factory method creational design pattern. Often, we find that some classes have common features (behavior) and can be considered classes of the same family. For example, multiple payment classes represent a family of payment services. Credit card, debit card, and net banking are some of the examples of payment classes that have common methods, such as makePayment, authorizePayment, and so on. Using the factory method pattern, we can develop controller classes, which can use these payment services, without knowing the actual payment type at design time. The factory method pattern is a creational design pattern used to create objects of classes from the same family without knowing the exact class name at design time. Using the factory method pattern, classes can be instantiated from the common factory method. The advantage of using this pattern is that it delegates the creation of an object to another class and provides a good level of abstraction. Let's learn this pattern using the following example: The Universal Call Center company is new in business and provides free admin support to customers to resolve issues related to their products. A call center agent can provide some information about the product support; for example, to get the Service Level Agreement (SLA) or information about the total number of tickets allowed to open per month. A developer came up with the following class: public class AdminBasicSupport{ /** * return SLA in hours */ public Integer getSLA() { return 40; } /** * Total allowed support tickets allowed every month */ public Integer allowedTickets() { // As this is basic support return 9999; } } Now, to get the SLA of AdminBasicSupport, we need to use the following code every time: AdminBasicSupport support = new AdminBasicSupport(); System.debug('Support SLA is - '+support.getSLA()); Output - Support SLA is – 40 The "Universal Call Centre" company was doing very well, and in order to grow the business and increase the profit, they started the premium support for customers who were willing to pay for cases and get a quick support. To make them special from the basic support, they changed the SLA to 12 hours and maximum 50 cases could be opened in one month. A developer had many choices to make this happen in the existing code. However, instead of changing the existing code, they created a new class that would handle only the premium support-related functionalities. This was a good decision because of the single responsibility principle. public class AdminPremiumSupport{ /** * return SLA in hours */ public Integer getSLA() { return 12; } /** * Total allowed support tickets allowed every month is 50 */ public Integer allowedTickets() { return 50; } } Now, every time any information regarding the SLA or allowed tickets per month is needed, the following Apex code can be used: if(Account.supportType__c == 'AdminBasic') { AdminBasicSupport support = new AdminBasicSupport(); System.debug('Support SLA is - '+support.getSLA()); }else{ AdminPremiumSupport support = new AdminPremiumSupport(); System.debug('Support SLA is - '+support.getSLA()); } As we can see in the preceding example, instead of adding some conditions to the existing class, the developer decided to go with a new class. Each class has its own responsibility, and they need to be changed for only one reason. If any change is needed in the basic support, then only one class needs to be changed. As we all know that this design principle is known as the Single Responsibility Principle. Business was doing exceptionally well in the call center, and they planned to start the golden and platinum support as well. Developers started facing issues with the current approach. Currently, they have two classes for the basic and premium support and requests for two more classes were in the pipeline. There was no guarantee that the support type will not remain the same in future. Because of every new support type, a new class is needed; and therefore, the previous code needs to be updated to instantiate these classes. The following code will be needed to instantiate these classes: if(Account.supportType__c == 'AdminBasic') { AdminBasicSupport support = new AdminBasicSupport(); System.debug('Support SLA is - '+support.getSLA()); }else if(Account.supportType__c == 'AdminPremier') { AdminPremiumSupport support = new AdminPremiumSupport(); System.debug('Support SLA is - '+support.getSLA()); }else if(Account.supportType__c == 'AdminGold') { AdminGoldSupport support = new AdminGoldSupport(); System.debug('Support SLA is - '+support.getSLA()); }else{ AdminPlatinumSupport support = new AdminPlatinumSupport(); System.debug('Support SLA is - '+support.getSLA()); } We are only considering the getSLA() method, but in a real application, there can be other methods and scenarios as well. The preceding code snippet clearly depicts the code duplicity and maintenance nightmare. The following image shows the overall complexity of the example that we are discussing: Although they are using a separate class for each support type, an introduction to a new support class will lead to changes in the code in all existing code locations where these classes are being used. The development team started brainstorming to make sure that the code is capable to extend easily in future with the least impact on the existing code. One of the developers came up with a suggestion to use an interface for all support classes so that every class can have the same methods and they can be referred to using an interface. The following interface was finalized to reduce the code duplicity: public Interface IAdminSupport{ Integer getSLA() ; Integer allowedTickets(); } Methods defined within an interface have no access modifiers and just contain their signatures. Once an interface was created, it was time to update existing classes. In our case, only one line needed to be changed and the remaining part of the code was the same because both the classes already have the getSLA() and allowedTickets() methods. Let's take a look at the following line of code: public class AdminPremiumSupport{ This will be changed to the following code: public class AdminBasicSupportImpl implements IAdminSupport{ The following line of code is as follows: public class AdminPremiumSupport{ This will be changed to the following code: public class AdminPremiumSupportImpl implements IAdminSupport{ In the same way, the AdminGoldSupportImpl and AdminPlatinumSupportImpl classes are written. A class diagram is a type of Unified Modeling Language (UML), which describes classes, methods, attributes, and their relationships, among other objects in a system. You can read more about class diagrams at https://en.wikipedia.org/wiki/Class_diagram. The following image shows a class diagram of the code written by developers using an interface: Now, the code to instantiate different classes of the support type can be rewritten as follows: IAdminSupport support = null; if(Account.supportType__c == 'AdminBasic') { support = new AdminBasicSupportImpl(); }else if(Account.supportType__c == 'AdminPremier') { support = new AdminPremiumSupportImpl(); }else if(Account.supportType__c == 'AdminGold') { support = new AdminGoldSupportImpl(); }else{ support = new AdminPlatinumSupportImpl(); } System.debug('Support SLA is - '+support.getSLA()); There is no switch case statement in Apex, and that's why multiple if and else statements are written. As per the product team, a new compiler may be released in 2016 and it will be supported. You can vote for this idea at https://success.salesforce.com/ideaView?id=08730000000BrSIAA0. As we can see, the preceding code is minimized to create a required instance of a concrete class, and then uses an interface to access methods. This concept is known as program to interface. This is one of the most recommended OOP principles suggested to be followed. As interfaces are kinds of contracts, we already know which methods will be implemented by concrete classes, and we can completely rely on the interface to call them, which hides their complex implementation and logic. It has a lot of advantages and a few of them are loose coupling and dependency injection. A concrete class is a complete class that can be used to instantiate objects. Any class that is not abstract or an interface can be considered a concrete class. We still have one problem in the previous approach. The code to instantiate concrete classes is still present at many locations and will still require changes if a new support type is added. If we can delegate the creation of concrete classes to some other class, then our code will be completely independent of the existing code and new support types. This concept of delegating decisions and creation of similar types of classes is known as the factory method pattern. The following class can be used to create concrete classes and will act as a factory: /** * This factory class is used to instantiate concrete class * of respective support type * */ public class AdminSupportFactory { public static IAdminSupport getInstance(String supporttype){ IAdminSupport support = null; if(supporttype == 'AdminBasic') { support = new AdminBasicSupportImpl(); }else if(supporttype == 'AdminPremier') { support = new AdminPremiumSupportImpl(); }else if(supporttype == 'AdminGold') { support = new AdminGoldSupportImpl(); }else if(supporttype == 'AdminPlatinum') { support = new AdminPlatinumSupportImpl(); } return support ; } } In the preceding code, we only need to call the getInstance(string) method, and this method will take a decision and return the actual implementation. As a return type is an interface, we already know the methods that are defined, and we can use the method without actually knowing its implementation. This is a very good example of abstraction. The final class diagram of the factory method pattern that we discussed will look like this: The following code snippet can be used repeatedly by any client code to instantiate a class of any support type: IAdminSupport support = AdminSupportFactory.getInstance ('AdminBasic'); System.debug('Support SLA is - '+support.getSLA()); Output : Support SLA is – 40 Reflection in Apex The problem with the preceding design is that whenever a new support needs to be added, we need to add a condition to AdminSupportFactory. We can store the mapping between a support type and its concrete class name in Custom setting. This way, whenever a new concrete class is added, we don't even need to change the factory class and a new entry needs to be added to custom setting. Consider custom setting created by the Support_Type__c name with the Class_Name__c field name of the text type with the following records: Name Class name AdminBasic AdminBasicSupportImpl AdminGolden AdminGoldSupportImpl AdminPlatinum AdminPlatinumSupportImpl AdminPremier AdminPremiumSupportImpl However, using reflection, the AdminSupportFactory class can also be rewritten to instantiate service types at runtime as follows: /** * This factory class is used to instantiate concrete class * of respective support type * */ public class AdminSupportFactory { public static IAdminSupport getInstance(String supporttype) { //Read Custom setting to get actual class name on basis of Support type Support_Type__c supportTypeInfo = Support_Type__c.getValues(supporttype); //from custom setting get appropriate class name Type t = Type.forName(supportTypeInfo.Class_Name__c); IAdminSupport retVal = (IAdminSupport)t.newInstance(); return retVal; } } In the preceding code, we are using the Type system class. This is a very powerful class used to instantiate a new class at runtime. It has the following two important methods: forName: This returns a type that is equivalent to a string passed newInstance: This creates a new object for a specified type Inspecting classes, methods, and variables at runtime without knowing a class name, or instantiating a new object and invoking methods at runtime is known as Reflection in computer science. One more advantage of using the factory method, custom setting, and reflection together is that if in future one of the support types need to be replaced by another service type permanently, then we need to simply change the appropriate mapping in custom setting without any changes in the code. Summary In this article, we discussed how to deal with various situations while instantiating objects using design patterns, using the factory method. Resources for Article: Further resources on this subject: Getting Your APEX Components Logic Right[article] AJAX Implementation in APEX[article] Custom Coding with Apex[article]
Read more
  • 0
  • 19
  • 34900

article-image-how-to-build-12-factor-design-microservices-on-docker-part-1
Cody A.
26 Jun 2015
9 min read
Save for later

How to Build 12 Factor Microservices on Docker - Part 1

Cody A.
26 Jun 2015
9 min read
As companies continue to reap benefits of the cloud beyond cost savings, DevOps teams are gradually transforming their infrastructure into a self-serve platform. Critical to this effort is designing applications to be cloud-native and antifragile. In this post series, we will examine the 12 factor methodology for application design, how this design approach interfaces with some of the more popular Platform-as-a-Service (PaaS) providers, and demonstrate how to run such microservices on the Deis PaaS. What began as Service Oriented Architectures in the data center are realizing their full potential as microservices in the cloud, led by innovators such as Netflix and Heroku. Netflix was arguably the first to design their applications to not only be resilient but to be antifragile; that is, by intentionally introducing chaos into their systems, their applications become more stable, scalable, and graceful in the presence of errors. Similarly, by helping thousands of clients building cloud applications, Heroku recognized a set of common patterns emerging and set forth the 12 factor methodology. ANTIFRAGILITY You may have never heard of antifragility. This concept was introduced by Nassim Taleb, the author of Fooled by Randomness and The Black Swan. Essentially, antifragility is what gains from volatility and uncertainty (up to a point). Think of the MySQL server that everyone is afraid to touch lest it crash vs the Cassandra ring which can handle the loss of multiple servers without a problem. In terms more familiar to the tech crowd, a “pet” is fragile while “cattle” are antifragile (or at least robust, that is, they neither gain nor lose from volatility). Adrian Cockroft seems to have discovered this concept with his team at Netflix. During their transition from a data center to Amazon Web Services, they claimed that “the best way to avoid failure is to fail constantly.” (http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html) To facilitate this process, one of the first tools Netflix built was Chaos Monkey, the now-infamous tool which kills your Amazon instances to see if and how well your application responds. By constantly injecting failure, their engineers were forced to design their applications to be more fault tolerant, to degrade gracefully, and to be better distributed so as to avoid any Single Points Of Failure (SPOF). As a result, Netflix has a whole suite of tools which form the Netflix PaaS. Many of these have been released as part of the Netflix OSS ecosystem. 12 FACTOR APPS Because many companies want to avoid relying too heavily on tools from any single third-party, it may be more beneficial to look at the concepts underlying such a cloud-native design. This will also help you evaluate and compare multiple options for solving the core issues at hand. Heroku, being a platform on which thousands or millions of applications are deployed, have had to isolate the core design patterns for applications which operate in the cloud and provide an environment which makes such applications easy to build and maintain. These are described as a manifesto entitled the 12-Factor App. The first part of this post walks through the first five factors and reworks a simple python webapp with them in mind. Part 2 continues with the remaining seven factors, demonstrating how this design allows easier integration with cloud-native containerization technologies like Docker and Deis. Let’s say we’re starting with a minimal python application which simply provides a way to view some content from a relational database. We’ll start with a single-file application, app.py. from flask import Flask import mysql.connector as db import json app = Flask(__name__) def execute(query): con = None try: con = db.connect(host='localhost', user='testdb', password='t123', database='testdb') cur = con.cursor() cur.execute(query) return cur.fetchall() except db.Error, e: print "Error %d: %s" % (e.args[0], e.args[1]) return None finally: if con: con.close() def list_users(): users = execute("SELECT id, username, email FROM users") or [] return [{"id": user_id, "username": username, "email": email} for (user_id, username, email) in users] @app.route("/users") def users_index(): return json.dumps(list_users()) if __name__ == "__main__": app.run(host='0.0.0.0', port=5000, debug=True) We can assume you have a simple mysql database setup already. CREATE DATABASE testdb; CREATE TABLE users ( id INT NOT NULL AUTO_INCREMENT, username VARCHAR(80) NOT NULL, email VARCHAR(120) NOT NULL, PRIMARY KEY (id), UNIQUE INDEX (username), UNIQUE INDEX (email) ); INSERT INTO users VALUES (1, "admin", "admin@example.com"); INSERT INTO users VALUES (2, "guest", "guest@example.com"); As you can see, the application is currently implemented as about the most naive approach possible and contained within this single file. We’ll now walk step-by-step through the 12 Factors and apply them to this simple application. THE 12 FACTORS: STEP BY STEP Codebase. A 12-factor app is always tracked in a version control system, such as Git, Mercurial, or Subversion. If there are multiple codebases, its a distributed system in which each component may be a 12-factor app. There are many deploys, or running instances, of each application, including production, staging, and developers' local environments. Since many people are familiar with git today, let’s choose that as our version control system. We can initialize a git repo for our new project. First ensure we’re in the app directory which, at this point, only contains the single app.py file. cd 12factor git init . After adding the single app.py file, we can commit to the repo. git add app.py git commit -m "Initial commit" Dependencies. All dependencies must be explicitly declared and isolated. A 12-factor app never depends on packages to be installed system-wide and uses a dependency isolation tool during execution to stop any system-wide packages from “leaking in.” Good examples are Gem Bundler for Ruby (Gemfile provides declaration and `bundle exec` provides isolation) and Pip/requirements.txt and Virtualenv for Python (where pip/requirements.txt provides declaration and `virtualenv --no-site-packages` provides isolation). We can create and use (source) a virtualenv environment which explicitly isolates the local app’s environment from the global “site-packages” installations. virtualenv env --no-site-packages source env/bin/activate A quick glance at the code we’ll show that we’re only using two dependencies currently, flask and mysql-connector-python, so we’ll add them to the requirements file. echo flask==0.10.1 >> requirements.txt echo mysql-python==1.2.5 >> requirements.txt Let’s use the requirements file to install all the dependencies into our isolated virtualenv. pip install -r requirements.txt Config. An app’s config must be stored in environment variables. This config is what may vary between deploys in developer environments, staging, and production. The most common example is the database credentials or resource handle. We currently have the host, user, password, and database name hardcoded. Hopefully you’ve at least already extracted this to a configuration file; either way, we’ll be moving them to environment variables instead. import os DATABASE_CREDENTIALS = { 'host': os.environ['DATABASE_HOST'], 'user': os.environ['DATABASE_USER'], 'password': os.environ['DATABASE_PASSWORD'], 'database': os.environ['DATABASE_NAME'] } Don’t forget to update the actual connection to use the new credentials object: con = db.connect(**DATABASE_CREDENTIALS) Backing Services. A 12-factor app must make no distinction between a service running locally or as a third-party. For example, a deploy should be able to swap out a local MySQL database with a third-party replacement such as Amazon RDS without any code changes, just by updating a URL or other handle/credentials inside the config. Using a database abstraction layer such as SQLAlchemy (or your own adapter) lets you treat many backing services similarly so that you can switch between them with a single configuration parameter. In this case, it has the added advantage of serving as an Object Relational Mapper to better encapsulate our database access logic. We can replace the hand-rolled execute function and SELECT query with a model object from flask.ext.sqlalchemy import SQLAlchemy app = Flask(__name__) app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL'] db = SQLAlchemy(app) class User(db.Model): __tablename__ = 'users' id = db.Column(db.Integer, primary_key=True) username = db.Column(db.String(80), unique=True) email = db.Column(db.String(120), unique=True) def __init__(self, username, email): self.username = username self.email = email def __repr__(self): return '<User %r>' % self.username @app.route("/users") def users_index(): to_json = lambda user: {"id": user.id, "name": user.username, "email": user.email} return json.dumps([to_json(user) for user in User.query.all()]) Now we set the DATABASE_URL environment property to something like export DATABASE_URL=mysql://testdb:t123@localhost/testdb But its should be easy to switch to Postgres or Amazon RDS (still backed by MySQL). DATABASE_URL=postgresql://testdb:t123@localhost/testdb We’ll continue this demo using a MySQL cluster provided by Amazon RDS. DATABASE_URL=mysql://sa:mypwd@mydbinstance.abcdefghijkl.us-west-2.rds.amazonaws.com/mydb As you can see, this makes attaching and detaching from different backing services trivial from a code perspective, allowing you to focus on more challenging issues. This is important during the early stages of code because it allows you to performance test multiple databases and third-party providers against one another, and in general keeps with the notion of avoiding vendor lock-in. In Part 2, we'll continue reworking this application so that it fully conforms to the 12 Factors. The remaining eight factors concern the overall application design and how it interacts with the execution environment in which its operated. We’ll assume that we’re operating the app in a multi-container Docker environment. This container-up approach provides the most flexibility and control over your execution environment. We’ll then conclude the article by deploying our application to Deis, a vertically integrated Docker-based PaaS, to demonstrate the tradeoff of configuration vs convention in selecting your own PaaS. About the Author Cody A. Ray is an inquisitive, tech-savvy, entrepreneurially-spirited dude. Currently, he is a software engineer at Signal, an amazing startup in downtown Chicago, where he gets to work with a dream team that’s changing the service model underlying the Internet.
Read more
  • 0
  • 0
  • 34595
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-chaos-engineering-managing-complexity-by-breaking-things
Richard Gall
20 Apr 2018
7 min read
Save for later

Chaos Engineering: managing complexity by breaking things

Richard Gall
20 Apr 2018
7 min read
Chaos Engineering is based on a fundamental assertion about software infrastructure today: that it is inherently chaotic. Or, to be more specific, it is chaotic because it is complex. Whereas software infrastructure used to be centralized, owned and licensed by large enterprise vendors, today much of the software that comprises infrastructure is open source. This is where we get back to chaos - because software infrastructure is comprised of many different parts, the way these parts can be unpredictable. Chaos Engineering is an attempt to acknowledge that fact and develop software accordingly. Who invented Chaos Engineering? Chaos Engineering began at Netflix. That makes sense when you consider the complexity of the Netflix technology stack and the way the company have scaled over the last 5 years or so. It built a number of tools to help adopt this chaos-first approach, the most prominent being Chaos Monkey. First launched in 2011 and open-sourced in 2012, Chaos Monkey was a tool that randomly selects instances in production and pulls them down; a little bit like monkeys pulling off your windscreen wipers in a safari park. However, Chaos Monkey became part of a wider suite of tools - called the Simian Army - that were built by Netflix to cause chaos in different part of its infrastructure. Here are the other two components used to simulate chaos: Chaos Gorilla causes big trouble by pulling down an entire AWS availability zone Latency monkey delays communication, essentially simulating poor network performance From that point Chaos Engineering grew. A number of large Silicon Valley organizations have adopted a similar approaches. For example, Facebook's Project Storm simulates data center failures on a huge scale, while Uber uses a tool called uDestroy. Slack has recently spoken in detail on the importance of stress testing their software too; the company is looking to build an engineering team simply to perform Chaos Engineering and improve Slack's reliability. One of the most interesting figures in Chaos Engineering is a man called Kolton Andrus. Andrus used to work at Amazon and Google, but today he is the CEO and founder of Gremlin, a startup that "helps engineers build resilient systems". Essentially, Andrus helped to develop the concept of Chaos Engineering while he was working at Netflix. Gremlin is his vehicle that is making it accessible to others. Chaos Engineering in practice Now the conceptual stuff is out of the way, here's how chaos engineering works. It's actually quite straightforward: Chaos Engineering simulates all sorts of unpredictable situations and scenarios in order to see how the system responds. It's effectively a form of stress testing. As we've seen, over the past few years companies have built their own tools to allow them to stress test their infrastructure. But Gremlin is taking the approach of offering this as a service. It's product is described as 'resiliency-as-a-service.' Its' product is a whole library of 'attacks' which can replicate different types of outages within a system. These are what it calls 'chaos experiments' that allows you to 'identify weak points in your system and fix them before they become a problem'. In this sense, Chaos Engineering is a bit like using the principles of penetration testing an applying it to software testing more broadly. By simulating everything that could possibly go wrong it allows you to make much better optimization decisions. The principles of Chaos Engineering are documented here. This is effectively its 'manifesto'. There's a lot in there worth reading, but here are the 5 principles that any sort of testing or experimentation should aspire to: Base your testing hypothesis on steady state behavior. Consider your infrastructure holistically, making individual parts work is important but not the priority. Simulate a variety of real-world events. This could be hardware or software failures, or simply external changes like spikes in traffic. What's important is that they're all unpredictable. Test in production. Your tests should be authentic. Automate! Testing things could be laborious and require a lot of manual work. Make use of automation tools to do lots of different tests without taking up too much of your time. Don't cause unnecessary pain. While it's important that your stress-tests are authentic, the impact must be contained and minimized by the engineer. Why Chaos Engineering now? Chaos Engineering isn't particularly new. As you've seen, Netflix has been doing it since 2011. But it does feel more urgent and relevant today. That's because the complexity of the software infrastructure behind many of the biggest Silicon Valley companies is now mainstream. It's normal. Cloud isn't an exotic buzzword any more - it's a reality (a reality that often has failures). Microservices are common - they're a commonsense way of building better applications and websites. Alongside this increased complexity, there is also a growing awareness of how much software outages can cost businesses. In a white paper, Gremlin make a big deal out of how much money is lost due to outages. Gremlin cite BAs system failure in summer 2017, which led to passengers stranded all over the world. This outage was estimated to have cost BA $135 million. It also refers to the Amazon S3 outage in March 2017, which is believed to have cost Amazon's customers $150 million. So - outages cost money. Yes, it's marketing spiel from Gremlin, but it's also true. It doesn't take a genius to work out that if you're eCommerce site is down for an hour, you're going to have lost a lot of money. Because software performance is so tied up with business performance, it feels incredibly fragile. That's why Chaos Engineering is perhaps more important and popular than ever. It's a way of countering that fragility. The key challenges of Chaos Engineering Chaos Engineering poses many challenges to software engineering teams. First and foremost, it requires a big cultural change. If you're intent on breaking everything, there are no rules about how things should work or what you're trying to build. Instead you're looking for the best way to build software that performs for the user. More practically, Chaos Engineering isn't that easy to do in a cost-effective manner. Everything Gremlin details in its white paper is very much true - of course outages cost a hell of a lot. But creative destruction and experimentation feels like an expensive route through software projects. It's not hard to see how it might appear self-indulgent, especially to a company or organization where software isn't properly understood. And more to the point, how often do businesses actually do the smart thing when they're building software? Long term projects are always difficult. So much software evolves pragmatically - often for the worse.  Adding in an extra layer of experimentation and detailed testing is a weird mix of bacchanalian and hyper-organized, something that many organizations just couldn't process or properly understand. Chaos engineering and the future of software development Chaos Engineering certainly looks like the future of software development. The only question is whether services like those provided by Gremlin will take off. To understand the true value of stress testing your infrastructure you do need at least a modicum awareness of the complexity of your infrastructure. Indeed, you probably need to have a conversation about what services and dependencies are most business critical. Or rather, which ones most impact the user. That's something this TechCrunch piece addresses: "Testing can... be very political. Finding the points of failure in a system might force deep conversations about a particular software architecture and its robustness in the face of tough situations. A particular company might be deeply invested in a specific technical roadmap (e.g. microservices) that chaos engineering tests show is not as resilient to failures as originally predicted." This means there is going to be a question mark over the extent to which Chaos Engineering ever really enters the mainstream. How many businesses want to have these conversations? It's not just about the inclination - it's also about the time and money. It's an innovative software engineering approach that really calls people's bluff when they talk about innovation. It asks difficult questions about how and why you innovate: do you do new things because you think you should? Is this new thing going to be good for the business? And how well will it work for users? Of course these questions are vital when you're building software. But they rarely make building software easier.
Read more
  • 0
  • 0
  • 34553

article-image-developers-think-managers-dont-know-about-technology
Fatema Patrawala
01 Jun 2018
7 min read
Save for later

Developers think managers don’t know enough about technology. And that’s hurting business.

Fatema Patrawala
01 Jun 2018
7 min read
It's not hard to find jokes online about management not getting software. There has long been a perception that those making key business decisions don't actually understand the technology and software that is at the foundation of just about every organization's operations. Now, research has confirmed that the management and engineering divide is real. In this year's Skill Up survey that we ran among 8000 developers, we found that more than 60% of developers believe they know more about technology than their manager. Source: Packtpub Skill Up Survey 2018 Developer perceptions on the topic aren't simply a question of ego; they're symptomatic of some a barriers to business success. 42% of the respondents listed management's lack of technical knowledge as a barrier to success. It also appears as one of the top 3 organizational barriers to achieving business goals. Source: Packtpub Skill Up Survey 2018 To dissect the technical challenges faced by organizations, we also asked respondents to pick the top technical barriers to success. As can be seen from the below graph, a lot of the barriers directly or indirectly relate to management’s understanding of technology. Take for example management’s decisions to continue with legacy systems, or investment or divestment from certain projects, choice of training programs and vendors etc. Source: Packtpub Skill Up Survey 2018 Management tends to weigh decisions based on the magnitude of investment against returns in immediate or medium term. Also, unless there is hard evidence of performance benefits or cost savings, management is generally wary of approving new projects or spending more on existing ones. This approach is generally robust and has saved business precious dollars by curbing pet projects and unruly experiments and research. However, with technology, things are always so straightforward. One day some tool is the talk of the town (think Adobe Flash) and everyone seems to be learning it or buying it and then in a few months or a couple of years down the line, it has gone completely off radar. Conversely, something that didn’t exist yesterday or was present in some obscure research lab (think self-driving tech, gene-editing, robotics etc), is now changing the rules of the game and businesses whose leadership teams have had their ears on the ground topple everyone else, including time tested veterans. Early adopters and early jumpers make the most of tech trends. This requires one (in the position to make decisions within organizations) to be aware of the changing tech landscape to the extent that one can predict what’s going to replace the current reigning tech and in what timeframe. It requires that management is aware of what’s happening in adjacent industries or even in seemingly unrelated ones. Who knew Unity (game platform), Nvidia (chipmaker), Google (search engine), would enter the auto industry, (all thanks to self driving tech)? While these are some over the top factors, let us look at each one of them in detail. Why do developers believe there is a management knowledge gap? Few reasons listed to justify the response: Rapid pace of technology change: The rapid rate of technology change is significantly impacting IT strategy. Not only are there plenty of emerging technology trends, from AI to cloud, they’re all coming at the same time, and even affecting each other. It’s clear that keeping up with the rate of digital advancement - for example automation, harnessing big data, emerging technologies and cyber security - will pose significant challenge for leaders and senior management. Adding a whole new layer of complexity as they try to stay ahead of competition and innovate. Balancing strategic priorities while complying to changing regulations: Another major challenge for senior management is to balance strategic priorities with the regulatory demands of the industry. In 2018, GDPR has been setting a new benchmark for the protection of consumer data rights by making organisations more accountable. Governed by GDPR, organisations and senior management will now be responsible for guarding every piece of information connected to an individual. In order to be GDPR compliant, management will begin introducing the correct security protocols in their business processes. This will include encryption, two-factor authentication and key management strategies to avoid severe legal, financial and reputational consequences.To make the right decisions, they will need to be technically competent enough to understand the strengths and limitations of the tools and techniques involved in the compliance process Finding right IT talent: Identifying the right talent with the skill sets that you need is a big challenge for senior management. They are constantly trying to find and hire IT talent, such as skilled data scientists and app developers, to accommodate and capitalize on emerging trends in cloud and the API economy. The team has to take care to bring in the right people and let them create magic with their development skills. Alongside this they also need to reinvent how they manage, retract, retain, motivate, and compensate these folks. Responses to this quora question highlight that it can be a difficult process for managers to go through a lengthy recruitment cycle. And the worst feeling is when after all the effort the candidate declines the offer for another lucrative one. So much promising technology, so little time: Time is tight in business and tech. Keeping pace with how quickly innovative and promising technologies crop up is easier said than done. There are so many interesting technologies out there, and there's so little time to implement them fast enough. Before anyone can choose a technology that might work for the company, a new product appears to be on the horizon. Once you see something you like, there's always something else popping up. While managers are working on a particular project to make all the parts work together for an outstanding customer experience, it requires time to do so and implement these technologies. When juggling with all of these moving parts, managers are always looking for technologies and ways to implement great things faster. That's the major reason behind companies having a CTO, VP of engineering and CEO separately to function at their own levels and departments. Murphy’s law of unforeseen IT problems: One of the biggest problems when you’re working in tech is Murphy's Law. This is the law that states  "Anything that can go wrong, will -- at the worst possible moment." It doesn't matter how hard we have worked, how strong the plan is, or how many times things are tested. You get to doing the project and if something has to go wrong, it will. There are times we face IT problems that we don't see coming. It doesn't matter how much you try to plan -- stuff happens. When management doesn’t properly understand technology it’s often hard for them to appreciate how problems arise and how long it can take to solve them. That puts pressure on engineers and developers which can make managing projects even harder. Overcoming perfectionism with an agile mindset: Senior management often wants things done yesterday, and they want it done perfectly. Of course, this is impossible. While Agile can help improve efficiency in the development process, perfectionism is anathema to Agile. It’s about delivering quickly and consistently, not building something perfect and then deploying it. Getting management to understand this is a challenge for engineers - good management teams will understand Agile and what the trade offs are. At the forefront of everyone’s mind should be what the customer needs and what is going to benefit the business. Concluding with Dilbert comic for a lighter note. Source With purpose, process, and changing technologies, managers need to change in the way they function and manage. People don't leave companies, they leave bad managers and the same could be said true for technical workers. They don't leave bad companies they leave non-technical managers who make bad technical decisions. Don’t call us ninjas or rockstars, say developers 96% of developers believe developing soft skills is important  
Read more
  • 0
  • 0
  • 34485

article-image-introduction-spring-framework
Packt
30 Dec 2016
10 min read
Save for later

Introduction to Spring Framework

Packt
30 Dec 2016
10 min read
In this article by, Tejaswini Mandar Jog, author of the book, Learning Spring 5.0, we will cover the following topics: Introduction to Spring framework Problems address by Spring in enterprise application development Spring architecture What's new in Spring 5.0 Container Spring the fresh new start after the winter of traditional J2EE, is what Spring framework is in actual. A complete solution to the most of the problems occurred in handling the development of numerous complex modules collaborating with each other in a Java enterprise application. Spring is not a replacement to the traditional Java Development but it is a reliable solution to the companies to withstand in today's competitive and faster growing market without forcing the developers to be tightly coupled on Spring APIs. Problems addressed by Spring Java Platform is long term, complex, scalable, aggressive, and rapidly developing platform. The application development takes place on a particular version. The applications need to keep on upgrading to the latest version in order to maintain recent standards and cope up with them. These applications have numerous classes which interact with each other, reuse the APIs to take their fullest advantage so as to make the application is running smoothly. But this leads to some very common problems of as. Scalability The growth and development of each of the technologies in market is pretty fast both in hardware as well as software. The application developed, couple of years back may get outdated because of this growth in these areas. The market is so demanding that the developers need to keep on changing the application on frequent basis. That means whatever application we develop today should be capable of handling the upcoming demands and growth without affecting the working application. The scalability of an application is handling or supporting the handling of the increased load of the work to adapt to the growing environment instead of replacing them. The application when supports handling of increased traffic of website due to increase in numbers of users is a very simple example to call the application is scalable. As the code is tightly coupled, making it scalable becomes a problem. Plumbing code Let's take an example of configuring the DataSource in the Tomcat environment. Now the developers want to use this configured DataSource in the application. What will we do? Yes, we will do the JNDI lookup to get the DataSource. In order to handle JDBC we will acquire and then release the resources in try catch. The code like try catch as we discuss here, inter computer communication, collections too necessary but are not application specific are the plumbing codes. The plumbing code increases the length of the code and makes debugging complex. Boilerplate code How do we get the connection while doing JDBC? We need to register Driver class and invoke the getConnection() method on DriverManager to obtain the connection object. Is there any alternative to these steps? Actually NO! Whenever, wherever we have to do JDBC these same steps have to repeat every time. This kind of repetitive code, block of code which developer write at many places with little or no modification to achieve some task is called as boilerplate code. The boilerplate code makes the Java development unnecessarily lengthier and complex. Unavoidable non-functional code Whenever application development happens, the developer concentrate on the business logic, look and feel and persistency to be achieved. But along with these things the developers also give a rigorous thought on how to manage the transactions, how to handle increasing load on site, how to make the application secure and many more. If we give a close look, these things are not core concerns of the application but still these are unavoidable. Such kind of code which is not handling the business logic (functional) requirement but important for maintenance, trouble shooting, managing security of an application is called as non-functional code. In most of the Java application along with core concerns the developers have to write down non-functional code quite frequently. This leads to provide biased concentration on business logic development. Unit testing of the application Let's take an example. We want to test a code which is saving the data to the table in database. Here testing the database is not our motive, we just want to be sure whether the code which we have written is working fine or not. Enterprise Java application consists of many classes, which are interdependent. As there is dependency exists in the objects it becomes difficult to carry out the testing. POJO based development The class is a very basic structure of application development. If the class is getting extended or implementing an interface of the framework, reusing it becomes difficult as they are tightly coupled with API. The Plain Old Java Object (POJO) is very famous and regularly used terminology in Java application development. Unlike Struts and EJB Spring doesn't force developers to write the code which is importing or extending Spring APIs. The best thing about Spring is that developers can write the code which generally doesn't has any dependencies on framework and for this, POJOs are the favorite choice. POJOs support loosely coupled modules which are reusable and easy to test. The Spring framework is called to be non-invasive as it doesn't force the developer to use API classes or interfaces and allows to develop loosely coupled application. Loose coupling through DI Coupling, is the degree of knowledge in class has about the other. When a class is less dependent on the design of any other class, the class will be called as loosely coupled. Loose coupling can be best achieved by interface programming. In the Spring framework, we can keep the dependencies of the class separated from the code in a separate configuration file. Using interfaces and dependency injection techniques provided by Spring, developers can write loosely coupled code (Don't worry, very soon we will discuss about Dependency Injection and how to achieve it). With the help of loose coupling one can write a code which needs a frequent change, due to the change in the dependency it has. It makes the application more flexible and maintainable. Declarative programming In declarative programming, the code states what is it going to perform but not how it will be performed. This is totally opposite of imperative programming where we need to state stepwise what we will execute. The declarative programming can be achieved using XML and annotations. Spring framework keeps all configurations in XML from where it can be used by the framework to maintain the lifecycle of a bean. As the development happened in Spring framework, the 2.0 onward version gave an alternative to XML configuration with a wide range of annotations. Boilerplate code reduction using aspects and templates We just have discussed couple of pages back that repetitive code is boilerplate code. The boiler plate code is essential and without which providing transactions, security, logging, and so on, will become difficult. The framework gives solution of writing aspect which will deal with such cross cutting concerns and no need to write them along with business logic code. The use of aspect helps in reduction of boilerplate code but the developers still can achieve the same end effect. One more thing the framework provides, is the templates for different requirements. The JDBCTemplate and HibernateTemplate are couple of more useful concepts given by Spring which does reduction of boilerplate code. But as a matter of fact, you need to wait to understand and discover the actual potential. Layered architecture Unlike Struts and Hibernate which provides web persistency solutions respectively, Spring has a wide range of modules for numerous enterprise development problems. This layered architecture helps the developer to choose any one or more of the modules to write solution for his application in a coherent way. E.g. one can choose Web MVC module to handle web request efficiently without even knowing that there are many other modules available in the framework. Spring architecture Spring provides more than 20 different modules which can be broadly summaries under 7 main modules which are as follows: Spring modules What more Spring supports underneath? The following sections covers the additional features of Spring. Security module Now a days the applications alone with basic functionalities also need to provide sound ways to handle security at different levels. Spring5 support declarative security mechanism using Spring AOP. Batch module The Java Enterprise Applications needs to perform bulk processing, handling of large amount of data in many business solutions without user interactions. To handle such things in batches is the best solution available. Spring provides integration of batch processing to develop robust application. Spring integration In the development of enterprise application, the application may need interaction with them. Spring integration is extension of the core spring framework to provide integration of other enterprise applications with the help of declarative adapters. The messaging is one of such integration which is extensively supported by Spring. Mobile module The extensive use of mobiles opens the new doors in development. This module is an extension of Spring MVC which helps in developing mobile web applications known as Spring Android Project. It also provide detection of the type of device which is making the request and accordingly renders the views. LDAP module The basic aim of Spring was to simplify the development and to reduce the boilerplate code. The Spring LDAP module supports easy LDAP integration using template based development. .NEW module The new module has been introduced to support .NET platform. The modules like ADO.NET, NHibernate, ASP.NET has been in the .NET module includes to simplify the .NET development taking the advantages of features as DI, AOP, loose coupling. Container – the heart of Spring POJO development is the backbone of Spring framework. The POJO configured in the and whose object instantiation, object assembly, object management is done by Spring IoC container is called as bean or Spring bean. We use Spring IoC as it on the pattern of Inversion of Control. Inversion of Control (IoC) In every Java application, the first important thing which each developer does is, to get an object which he can use in the application. The state of an object can be obtained at runtime or it may be at compile time. But developers creates object where he use boiler plate code at a number of times. When the same developer uses Spring instead of creating object by himself he will be dependent on the framework to obtain object from. The term inversion of control comes as Spring container inverts the responsibility of object creation from developers. Spring IoC container is just a terminology, the Spring framework provides two containers: The BeanFactory The ApplicationContext Summary So in this article, we discussed about the general problems faced in Java enterprise application development and how they have been address by Spring framework. We have seen the overall major changes happened in each version of Spring from its first introduction in market. Enabling Spring Faces support Design with Spring AOP Getting Started with Spring Security
Read more
  • 0
  • 0
  • 34406

article-image-pycon-2019-highlights-python-steering-council-discusses-the-changes-in-the-current-python-governance-structure
Bhagyashree R
07 May 2019
8 min read
Save for later

PyCon 2019 highlights: Python Steering Council discusses the changes in the current Python governance structure

Bhagyashree R
07 May 2019
8 min read
At the ongoing PyCon 2019 event, Python Steering Council shed some light on the recent changes in the Python governance structure and what these changes mean for the larger Python community. PyCon 2019 is the biggest gathering of developers and experts who work with the Python programming language. It is scheduled from May 1 to May 9 and is happening at Cleveland, Ohio. Backed by Python Software Foundation (PSF), this event hosts various tutorials, talks, summits, as well as a job fair. The Python Steering Council After a two week nomination period (January 7 to January 20), which was followed by a two week voting period (January 21 to February 4), five members were selected for the Python Steering Council: Guido van Rossum, the brilliant mind behind the Python programming language and the former Python BDFL (Benevolent Dictator for Life). Barry Warsaw, a Senior Staff Software Engineer at LinkedIn, and also the lead maintainer for Jython. Brett Cannon, a Principal Software Engineering Manager at Microsoft and a  Python core developer for over 15 years. Carol Willing, a Research Software Engineer for Project Jupyter, Python core developer, and PSF Fellow Nick Coghlan, a CPython Core developer for Python Software Foundation On why Guido van Rossum stepped down from being BDFL Since the dawn of the Python era, Guido van Rossum served as its Benevolent Dictator for Life (BDFL). It is a designation given to open-source software development leaders who have the final say in any kind of argument within the community. Guido stepped down from this designation last year in July and became a part of the Steering Council. Being a BDFL he was responsible for going through all the Python-ideas that might become controversial. Eventually, it ended up becoming his responsibility to take the final decision for PEPs which has already been discussed among the people with greater domain knowledge and expertise. After playing such a key authoritative role for nearly 30 years, Guido started experiencing, what is really common nowadays in the tech industry, the burnout syndrome. So, he finally took the right step of stepping down from his role as a BDFL and urging the core python core developers to discuss and decide amongst themselves the kind of governance structure they want for the community going forward. After months of intense research and debate, the team arrived at the decision of distributing these responsibilities among the five elected steering council members who have earned the trust of the Python community. He adds, “...that's pretty stressful and so I'm very glad that responsibility is now distributed over five experts who have more trust of the community because they've actually been voted in rather than just becoming the leader by happenstance.” Sharing his feelings about stepping down from the BDFL role, he said, “...when your kid goes off to college some of you may have experience with that I will soon have that experience. You're no longer directly involved in their lives maybe but you never stop worrying and that's how I feel about Python at the moment and that's why I nominated myself for the steering committee.” Changes in the PEP process with the new governance model The purpose behind Python Enhancement Proposals (PEPs) was to take away the burden from Guido of going through each and every email to understand what the proposal was about. He just needed to read one document listing all the pros and cons related to the proposal and then make a decision. This entire decision-making process was documented within the PEPs. With the growing Python community, this process became quite unattainable for Guido as all the decisions funneled through him. So, that is why the idea of BDFL delegate came up: an expert who will take care of the decision-making for a particular feature. However, earlier employing a BDFL delegate was the last resort and it was done for those aspects of the ecosystem that Guido didn't want to get involved in. With the new governance model, this has become the first resort. Barry Warsaw, said, “...we don't want to make those decisions if there are people in the community who are better equipped to do that. That's what we want to do, we want to allow other people to become engaged with shaping where Python is going to go in the next 25 years.” Hiring a Project Manager to help transition from Python 2 to 3 The countdown for Python 2 has started and it will not be maintained past 2019. The steering council has plans for hiring a Project Manager to help them better manage the sunset of Python 2. The PM will also have the responsibility of looking into minor details as well, for instance, in the documentation, there is mention of Python 2 and 3. These instances will need to be updated, as from 2020 we will only have Python and eventually developers will not have to care about the major numbers. For the systems that haven't migrated, there will be commercial vendors offering support beyond 2020. There will also be options for business-critical systems, but it will take time, shared Willing. One of the responsibilities that the PM role will take care of will be looking into the various best practices that other companies have followed for the migration and help others to easily migrate. Willing said, “Back in a couple of years, Instagram did a great keynote about how they were moving things from 2 to 3. I think one of the things that we want a PM to help us in this transition is to really take those best practices that we're learning from large companies who have found the business case to transition to make it easier.” Status of CPython issue tracking migration from Roundup to GitHub All PSF’s projects including CPython have moved to GitHub, but the issue tracking for CPython is still done through Roundup. Marieta Wijaya, a Platform Engineer at Zapier and a core python developer, wrote PEP 581 that proposes using GitHub for its issue tracking. Barry Warsaw has taken the initial steps and split the PEP into PEP 581 and 588. While PEP 581 gives the rationale and background, PEP 588 gives a detailed plan of how the migration will take place. The council has requested the PSF to hire a PM to take the responsibilities of the migration. Brett Cannon adds, “...with even the PSF about potentially trying to have a PM sort of role to help handle the migration because we realize that if we go forward with this the migration of those issues are going to be critical and we don't want any problems.” The features or improvements Python Packaging Workgroup should now focus on The Python Packaging Workgroup supports the efforts taken for improving and maintaining the packaging ecosystem in Python by fundraising and distributing this fund among different efforts. The efforts this workgroup supports include PyPI, pip, packaging.python.org, setuptools, and cross-project efforts. Currently, the workgroup is supporting the Warehouse project, which is a new implementation of PyPI aiming to solve the issues PyPI users face. Last year, the workgroup came out with the Warehouse code base and in March this year they have laid out work for the next set of improvements which will be around security and accessibility. When Coghlan was asked about what are the next steps now, he shared that they are looking into improving the overall publisher experience. He adds, that though there have been improvements in the consumer experience, very fewer efforts have been put on improving the publisher side. Publisher-side releases are becoming complicated, people want to upload source distributions with multiple wheels for different platforms and different Python versions. Currently, the packaging process is not that flexible. “at the moment the packing index is kind of this instant publish thing like you push it up and it's done...we'd really like to be able to offer a staging area where people can put up all the artifacts for release, make sure everything's in order, and then once they're happy with the release, push button and have it go publish.” These were some of the highlights from the discussion about the changes in the Python governance structure. You can watch the full discussion on YouTube: https://www.youtube.com/watch?v=8dDp-UHBJ_A&feature=youtu.be&t=379 Creators of Python, Java, C#, and Perl discuss the evolution and future of programming language design at PuPPy Mozilla introduces Pyodide, a Python data science stack compiled to WebAssembly RStudio 1.2 releases with improved testing and support for Python chunks, R scripts, and much more!
Read more
  • 0
  • 0
  • 34349
article-image-bitbucket-to-no-longer-support-mercurial-users-must-migrate-to-git-by-may-2020
Fatema Patrawala
21 Aug 2019
6 min read
Save for later

Bitbucket to no longer support Mercurial, users must migrate to Git by May 2020

Fatema Patrawala
21 Aug 2019
6 min read
Yesterday marked an end of an era for Mercurial users, as Bitbucket announced to no longer support Mercurial repositories after May 2020. Bitbucket, owned by Atlassian, is a web-based version control repository hosting service, for source code and development projects. It has used Mercurial since the beginning in 2008 and then Git since October 2011. Now almost after ten years of sharing its journey with Mercurial, the Bitbucket team has decided to remove the Mercurial support from the Bitbucket Cloud and its API. The official announcement reads, “Mercurial features and repositories will be officially removed from Bitbucket and its API on June 1, 2020.” The Bitbucket team also communicated the timeline for the sunsetting of the Mercurial functionality. After February 1, 2020 users will no longer be able to create new Mercurial repositories. And post June 1, 2020 users will not be able to use Mercurial features in Bitbucket or via its API and all Mercurial repositories will be removed. Additionally all current Mercurial functionality in Bitbucket will be available through May 31, 2020. The team said the decision was not an easy one for them and Mercurial held a special place in their heart. But according to a Stack Overflow Developer Survey, almost 90% of developers use Git, while Mercurial is the least popular version control system with only about 3% developer adoption. Apart from this Mercurial usage on Bitbucket saw a steady decline, and the percentage of new Bitbucket users choosing Mercurial fell to less than 1%. Hence they decided on removing the Mercurial repos. How can users migrate and export their Mercurial repos Bitbucket team recommends users to migrate their existing Mercurial repos to Git. They have also extended support for migration, and kept the available options open for discussion in their dedicated Community thread. Users can discuss about conversion tools, migration, tips, and also offer troubleshooting help. If users prefer to continue using the Mercurial system, there are a number of free and paid Mercurial hosting services for them. The Bitbucket team has also created a Git tutorial that covers everything from the basics of creating pull requests to rebasing and Git hooks. Community shows anger and sadness over decision to discontinue Mercurial support There is an outrage among the Mercurial users as they are extremely unhappy and sad with this decision by Bitbucket. They have expressed anger not only on one platform but on multiple forums and community discussions. Users feel that Bitbucket’s decision to stop offering Mercurial support is bad, but the decision to also delete the repos is evil. On Hacker News, users speculated that this decision was influenced by potential to market rather than based on technically superior architecture and ease of use. They feel GitHub has successfully marketed Git and that's how both have become synonymous to the developer community. One of them comments, “It's very sad to see bitbucket dropping mercurial support. Now only Facebook and volunteers are keeping mercurial alive. Sometimes technically better architecture and user interface lose to a non user friendly hard solutions due to inertia of mass adoption. So a lesson in Software development is similar to betamax and VHS, so marketing is still a winner over technically superior architecture and ease of use. GitHub successfully marketed git, so git and GitHub are synonymous for most developers. Now majority of open source projects are reliant on a single proprietary solution Github by Microsoft, for managing code and project. Can understand the difficulty of bitbucket, when Python language itself moved out of mercurial due to the same inertia. Hopefully gitlab can come out with mercurial support to migrate projects using it from bitbucket.” Another user comments that Mercurial support was the only reason for him to use Bitbucket when GitHub is miles ahead of Bitbucket. Now when it stops supporting Mercurial too, Bitbucket will end soon. The comment reads, “Mercurial support was the one reason for me to still use Bitbucket: there is no other Bitbucket feature I can think of that Github doesn't already have, while Github's community is miles ahead since everyone and their dog is already there. More importantly, Bitbucket leaves the migration to you (if I read the article correctly). Once I download my repo and convert it to git, why would I stay with the company that just made me go through an annoying (and often painful) process, when I can migrate to Github with the exact same command? And why isn't there a "migrate this repo to git" button right there? I want to believe that Bitbucket has smart people and that this choice is a good one. But I'm with you there - to me, this definitely looks like Bitbucket will die.” On Reddit, programming folks see this as a big change from Bitbucket as they are the major mercurial hosting provider. And they feel Bitbucket announced this at a pretty short notice and they require more time for migration. Apart from the developer community forums, on Atlassian community blog as well users have expressed displeasure. A team of scientists commented, “Let's get this straight : Bitbucket (offering hosting support for Mercurial projects) was acquired by Atlassian in September 2010. Nine years later Atlassian decides to drop Mercurial support and delete all Mercurial repositories. Atlassian, I hate you :-) The image you have for me is that of a harmful predator. We are a team of scientists working in a university. We don't have computer scientists, we managed to use a version control simple as Mercurial, and it was a hard work to make all scientists in our team to use a version control system (even as simple as Mercurial). We don't have the time nor the energy to switch to another version control system. But we will, forced and obliged. I really don't want to check out Github or something else to migrate our projects there, but we will, forced and obliged.” Atlassian Bitbucket, GitHub, and GitLab take collective steps against the Git ransomware attack Attackers wiped many GitHub, GitLab, and Bitbucket repos with ‘compromised’ valid credentials leaving behind a ransom note BitBucket goes down for over an hour
Read more
  • 0
  • 0
  • 34327

article-image-rookout-and-appdynamics-team-up-to-help-enterprise-engineering-teams-debug-at-speed-with-deep-code-insights
Richard Gall
20 Feb 2020
3 min read
Save for later

Rookout and AppDynamics team up to help enterprise engineering teams debug at speed with Deep Code Insights

Richard Gall
20 Feb 2020
3 min read
It's not acknowledged enough that the real headache when it comes to software faults and performance problems isn't so much the problems themselves, but instead the process of actually identifying those problems. Sure, problems might slow you down, but wading though your application code to actually understand what's happened can sometimes grind engineering teams to a halt. For enterprise engineering teams, this can be particularly fatal. Agility is hard enough when you're dealing with complex applications and the burden of legacy software; but when things go wrong, any notion of velocity can be summarily discarded to the trashcan. However, a new partnership between debugging platform Rookout and APM company AppDynamics, announced at AppDynamics' Transform 2020 event, might just change that. The two organizations have teamed up, with Rookout's impressive debugging capabilities now available to AppDynamics customers in the form of a new product called Deep Code Insights. [caption id="attachment_31042" align="alignleft" width="696"] Live debugging of an application in production in Deep Code Insights[/caption]                 What is Deep Code Insights? Deep Code Insights is a new product for AppDynamics customers that combines the live-code debugging capabilities offered by Rookout with AppDynamic's APM platform. The advantage for developers could be substantial. Jerrie Pineda, Enterprise Software Architect at Maverik says that "Rookout helps me get the debugging data I need in seconds instead of waiting for several hours." This means, he explains, "[Maverik's] mean time to resolution (MTTR) for most issues is slashed up to 80%.” What does Deep Code Insights mean for AppDynamics? For AppDynamics, Deep Code Insights allows the organization to go one step further in its mission to "make it easier for businesses to understand their own software." At least that's how AppDynamics' VP of corporate development and strategy at Kevin Wagner puts it. "Together [with Rookout], we are narrowing the gaps between indicating a code-related problem impacting performance, pinpointing the direct issue within the line of code, and deploying a solution quickly for a seamless customer experience," he says. What does Deep Code Insights mean for Rookout? For Rookout, meanwhile, the partnership with AppDynamics is a great way for the company to reach out to a wider audience of users working at large enterprise organizations. The company received $8,000,000 in Series A funding back in August. This has provided a solid platform on which it is clearly looking to build and grow. Rookout's Co-Founder and CEO Or Weis describes the partnership as "obvious." "We want to bring the next-gen developer workflow to enterprise customers and help them increase product velocity," he says. Learn more about Rookout: www.rookout.com Learn more about AppDynamics: www.appdynamics.com  
Read more
  • 0
  • 0
  • 34131

article-image-the-v-programming-language-is-now-open-sourced-is-it-too-good-to-be-true
Bhagyashree R
24 Jun 2019
5 min read
Save for later

The V programming language is now open source - is it too good to be true?

Bhagyashree R
24 Jun 2019
5 min read
Yesterday, a new statically-typed programming language named V was open sourced. It is described as a simple, fast, and compiled language for creating maintainable software. Its creator, Alex Medvednikov, says that it is very similar to Go and is inspired by Oberon, Rust, and Swift. What to expect from V programming language Fast compilation V can compile up to 1.2 million lines of code per second per CPU. It achieves this by direct machine code generation and strong modularity. If we decide to emit C code, the compilation speed drops to approximately 100k of code per second per CPU. Medvednikov mentions that direct machine code generation is still in its very early stages and right now only supports x64/Mach-O. He plans to make this feature stable by the end of this year. Safety It seems to be an ideal language because it has no null, global variables, undefined values, undefined behavior, variable shadowing, and does bound checking. It supports immutable variables, pure functions, and immutable structs by default. Generics are right now work in progress and are planned for next month. Performance According to the website, V is as fast as C, requires a minimal amount of allocations, and supports built-in serialization without runtime reflection. It compiles to native binaries without any dependencies. Just a 0.4 MB compiler Compared to Go, Rust, GCC, and Clang, the space required and build time of V are very very less. The entire language and standard library is just 400 KB and you can build it in 0.4s. By the end of this year, the author aims to bring this build time down to 0.15s. C/C++ translation V allows you to translate your V code to C or C++. However, this feature is at a very early stage, given that C and C++ are a very complex language. The creator aims to make this feature stable by the end of this year. What do developers think about this language? As much as developers like to have a great language to build applications, many felt that V is too good to be true. Looking at the claims made on the site some developers thought that the creator is either not being truthful about the capabilities of V or is scamming people. https://twitter.com/warnvod/status/1112571835558825986 A language that has the simplicity of Go and the memory management model of Rust is what everyone desires. However, the main reason that makes people skeptical about V is that there is not much proof behind the hard claims it makes. A user on Hacker news commented, “...V's author makes promises and claims which are then retracted, falsified, or untestable. Most notably, the source for V's toolchain has been teased repeatedly as coming soon but has never been released. Without an open toolchain, none of the claims made on V's front page [2] can be verified.” Another thing that makes this case concerning is that the V programming language is currently in alpha stage and is incomplete. Despite that, the creator is making $827 per month from his Patreon account. “However, advertising a product can do something and then releasing it stating it cannot do it yet, is one thing, but accepting money for a product that does not what is advertised, is a fraud,” a user commented. Some developers are also speculating that the creator is maybe just embarrassed to open source his code because of bad coding pattern choices. A user speculates, “V is not Free Software, which is disappointing but not atypical; however, V is not even open source, which precludes a healthy community. Additionally, closed languages tend to have bad patterns like code dumps over the wall, poor community communication, untrustworthy binary behaviors, and delayed product/feature releases. Yes, it's certainly embarrassing to have years of history on display for everybody to see, but we all apparently have gotten over it. What's hiding in V's codebase? We don't know. As a best guess, I think that the author may be ashamed of the particular nature of their bootstrap.” The features listed on the official website are incredible. The only concern was that the creator was not being transparent about how he plans to achieve them. Also, as this was closed source earlier, there was no way for others to verify the performance guarantees it promises that’s why so much confusion happened. Alex Medvednikov on why you can trust V programming On an issue that was reported on GitHub, the creator commented, “So you either believe me or you don't, we'll see who is right in June. But please don't call me a liar, scammer and spread misinformation.” Medvednikov was maybe overwhelmed by the responses and speculations, he was seeing on different discussion forums. Developing a whole new language requires a lot of work and perhaps his deadlines are ambitious. Going by the release announcement Medvednikov made yesterday, he is aware that the language designing process hasn’t been the most elegant version of his vision. He wrote, “There are lots of hacks I'm really embarrassed about, like using os.system() instead of native API calls, especially on Windows. There's a lot of ugly C code with #, which I regret adding at all.” Here’s great advice shared by a developer on V’s GitHub repository: Take your time, good software takes time. It's easy to get overwhelmed building Free software: sometimes it's better to say "no" or "not for now" in order to build great things in the long run :) Visit the official website of the V programming language for more detail. Docker and Microsoft collaborate over WSL 2, future of Docker Desktop for Windows is near Pull Panda is now a part of GitHub; code review workflows now get better! Scala 2.13 is here with overhauled collections, improved compiler performance, and more!
Read more
  • 0
  • 0
  • 34089
article-image-style-management-qgis
Packt
06 Jul 2015
11 min read
Save for later

Style Management in QGIS

Packt
06 Jul 2015
11 min read
In this article by Alexander Bruy and Daria Svidzinska, authors of the book QGIS By Example, you will learn how to work with styles, including saving and loading them, using different styles, and working with the Style Manager. (For more resources related to this topic, see here.) Working with styles In QGIS, a style is a way of cartographic visualization that takes into account a layer’s individual and thematic features. It encompasses basic characteristics of symbology, such as the color and presence of fill, outline parameters, the use of markers, scale-dependent rendering, layer transparency, and interactions with other layers. Style incorporates not only rendering appearance, but also other things, such as labeling settings. A well-chosen style greatly simplifies data perception and readability, so it is important to learn how to work with styles to be able to represent your data the best way. Styling is an important part of data preparation, and QGIS provides many handy features that make this process much more productive and easier. Let’s look at some of them! Saving and loading styles Creating good-looking styles can be a time-consuming task, but the good thing is that once developed styles don’t go to nowhere. You can save them for further use in other projects. When you have finished polishing your style, it is wise to save it. Usually, this is done from the Layer Properties dialog, which can be opened from the layer's context menu. There is a Style button at the bottom of this dialog. It provides access to almost all actions that can be performed with the layer's style, including saving, loading, making the style default, and so on. The style can be saved to the file on the disk (this works for any layer type), or stored in the database (possible only for database-based layers). To save style in the file, perform these steps: Open the Layer Properties dialog. Click on the Style button at the bottom of the Properties dialog and go to the Save Style submenu: Choose one of the available formats. A standard file dialog will open. Navigate to the desired location in your filesystem and save the style. Currently QGIS provides support for the following formats of saving styles: QGIS style file: The style is saved as a .qml file, which is a native QGIS format used to store symbology definition and other layer properties. Style Layer Descriptor (SLD) file: The style is exported to a .sld file. SLD format is widely used in web cartography, for example, by applications such as GeoServer. It is necessary to mention that currently, SLD support in QGIS is a bit limited. Also, you should remember that while you can save any style (or renderer type) in SLD, during import, you will get either a rule-based or a single-symbol renderer. If you work with the spatial database, you may want to save layer styles in the same database, together with layers. Such a feature is very useful in corporate environments, as it allows you to assign multiple styles for a single layer and easily keep the styles in sync. Saving styles in the database currently works only for PostgreSQL and SpatiaLite. To save style in the database, follow these steps: Open the Layer Properties dialog. Click on the Style button at the bottom of the Properties dialog and go to the Save Style submenu. Select the Save in database (format) item, where format can be spatialite or postgres, depending on the database type: The Save style in database dialog opens. Enter the style name and (optional) description in the corresponding fields, and click on the OK button to save the style: The saved style can be loaded and applied to the layer. To load a style from the file, use these steps: Open the Layer Properties dialog from the context menu. Click on the Style button at the bottom of the Properties dialog and select the Load Style item. Choose the style file to load. Loading a style from the database is a bit different: Open the Layer Properties dialog from the context menu. Click on the Style button at the bottom of the Properties dialog and go to Load Style | From database. The Load style from database dialog opens. Select the style you want to load and click on the Load Style button. With all of these options, we can easily save styles in the format that meets our requirements and tasks. Copy and paste styles Very often, you need to apply mostly the same style with really minor differences to multiple layers. There are several ways of doing this. First, you can save the style (as described in the previous section) in one of the supported formats, and then apply this saved style to another layer and edit it. But there is simpler way. Starting from QGIS 1.8, you can easily copy and paste styles between layers. To copy a style from one layer to another, perform these steps: In the QGIS layer tree, select the source layer from which you want to copy the style. Right-click to open the context menu. Go to Styles | Copy Style to copy the style of the source layer to the clipboard. Now, in the QGIS layer tree, select the target layer. Right-click to open its context menu. Go to Styles | Paster Style to paste the previously copied style from the clipboard and apply it to the target layer. It is important to note that QGIS allows you to copy, for example, a polygonal style and apply it to a point or line layer. This may lead to incorrect layer rendering, or the layer can even disappear from the map even though it still present in the layer tree. Instead of using the layer context menu to copy and paste styles, you can use the QGIS main menu. Both of these actions (Copy Style and Paste Style) can be found in the Layer menu. The copied style can be pasted in a text editor. Just copy the style using the context menu or QGIS main menu, open the text editor, and press Ctrl + V (or another shortcut used in your system to paste data from the clipboard) to paste the style. Now you can study it. Also, with this feature, you can apply the same style to multiple layers at once. Copy the style as previously described. Then select in the QGIS layer tree all the layers that you want to style (use the Ctrl key and/or the Shift key to select multiple layers). When all the desired layers are selected, go to Layer | Paste Style. Voilà! Now the style is applied to all selected layers. Using multiple styles per layer Sometimes, you may need to show the same data with different styles on the map. The most common and well-known solution to do this is to duplicate the layer in the QGIS layer tree and change the symbology. QGIS 2.8 allows us to achieve the same result in a simpler and more elegant way. Now we can define multiple styles for a single layer and easily switch between them when necessary. This functionality is available from the layer context menu and the layer Properties dialog. By default, all layers have only one style, called default. To create an additional style, use these steps: Select the layer in the layer tree. Right-click to open the context menu. Go to Styles | Add. The New style dialog opens. Enter the name of the new style and click on OK: A new style will be added and become active. It is worth mentioning that after adding a new style, the layer's appearance will remain the same, as the new style inherits all the properties of the active style. Adjust the symbology and other settings according to your needs. These changes will affect only the current style; previously created styles will remain unchanged. You can add as many styles as you want. All available styles are listed in the layer context menu, at the bottom of the Styles submenu. The current (or active) style is marked with a checkbox. To switch to another style, just select its name in the menu. If necessary, you can rename the active style (go to Styles | Rename Current) or remove it (go to Styles | Remove Current). Also, the current style can be saved and copied as previously described. Moreover, it is worth mentioning that multiple styles are supported by the QGIS server. The available layer styles are displayed via the GetCapabilities response, and the user can request them in, for example, the GetMap request. This handy feature also works in the Print Composer. Using Style manager Style manager provides extended capabilities for symbology management, allowing the user to save developed symbols; tag and merge them into thematic groups; and edit, delete, import, or export ready-to-use predefined symbology sets. If you created a symbol and want it to be available for further use and management, you should first save it to the Symbol library by following these steps: In the Style section of the layer Properties window, click on the Save button underneath the symbol preview window. In the Symbol name window, type a name for the new symbol and click on OK: After that, the symbol will appear and become available from the symbol presets in the right part of the window. It will also become available for the Style Manager. The Style Manger window can be opened by: Clicking on the Open library button after going to Properties | Style Going to Settings | Style Manager The window consists of three sections: In the left section, you can see a tree view of the available thematic symbology groups (which, by default, don't contain any user-specified groups) In the right part, there are symbols grouped on these tabs: Marker (for point symbols), Line (for line symbols), Fill (for polygon symbols), and Color ramp (for gradient symbols). If you double-click on any symbol on these tabs, the Symbol selector window will be opened, where you can change any available symbol properties (Symbol layer type, Size, Fill and Outline colors, and so on). Similarly, you can use the Edit button to change the appearance of the symbol. The bottom section of the window contains symbol management buttons—Add, Remove, Edit, and Share—for groups and their items. Let's create a thematic group called Hydrology. It will include symbology for hydrological objects, whether they are linear (river, canal, and so on) or polygonal (lake, water area, and so on). For this, perform the following steps: Highlight the groups item in the left section of the Style manager window and click on the very first + button. When the New group appears, type the Hydrology name. Now you need to add some symbols to the newly emerged group. There are two approaches to doing this: Right-click on any symbol (or several by holding down the Ctrl key) you want to add, and select from its contextual shortcut Apply Group | Hydrology. Alternatively, highlight the Hydrology group in the groups tree, and from the button below, select Group Symbols, as shown in this screenshot: As a result, checkboxes will appear beside the symbols, and you can toggle them to add the necessary symbol (or symbols) to the group. After you have clicked on the Close button, the symbols will be added to the group. Once the group is created, you can use it for quick access to the necessary symbology by going to Properties | Style | Symbols in group, as shown in the following screenshot: Note that you can combine the symbology for different symbology types within a single group (Marker, Line, Fill, and Color ramp), but when you upload symbols in this group for a specific layer, the symbols will be filtered according to the layer geometry type (for example, Fill for the polygon layer type). Another available option is to create a so-called Smart Group, where you can flexibly combine various conditions to merge symbols into meaningful groups. As an example, the following screenshot shows how we can create a wider Water group that includes symbols that are not only present in Hydrology already, but are also tagged as blue: Use the Share button to Export or Import selected symbols from external sources. Summary This article introduced the different aspects of style management in QGIS: saving and loading styles, copying and pasting styles, and using multiple styles per layer. Resources for Article: Further resources on this subject: How Vector Features are Displayed [article] Geocoding Address-based Data [article] Editing attributes [article]
Read more
  • 0
  • 0
  • 33960

article-image-collaboration-using-github-workflow
Packt
30 Sep 2015
12 min read
Save for later

Collaboration Using the GitHub Workflow

Packt
30 Sep 2015
12 min read
In this article by Achilleas Pipinellis, the author of the book GitHub Essentials, has come up with a workflow based on the features it provides and the power of Git. It has named it the GitHub workflow (https://guides.github.com/introduction/flow). In this article, we will learn how to work with branches and pull requests, which is the most powerful feature of GitHub. (For more resources related to this topic, see here.) Learn about pull requests Pull request is the number one feature in GitHub that made it what it is today. It was introduced in early 2008 and is being used extensively among projects since then. While everything else can be pretty much disabled in a project's settings (such as issues and the wiki), pull requests are always enabled. Why pull requests are a powerful asset to work with Whether you are working on a personal project where you are the sole contributor or on a big open source one with contributors from all over the globe, working with pull requests will certainly make your life easier. I like to think of pull requests as chunks of commits, and the GitHub UI helps you visualize clearer what is about to be merged in the default branch or the branch of your choice. Pull requests are reviewable with an enhanced diff view. You can easily revert them with a simple button on GitHub and they can be tested before merging, if a CI service is enabled in the project. The connection between branches and pull requests There is a special connection between branches and pull requests. In this connection, GitHub will automatically show you a button to create a new pull request if you push a new branch in your repository. As we will explore in the following sections, this is tightly coupled to the GitHub workflow, and GitHub uses some special words to describe the from and to branches. As per GitHub's documentation: The base branch is where you think changes should be applied, the head branch is what you would like to be applied. So, in GitHub terms, head is your branch, and base the branch you would like to merge into. Create branches directly in a project – the shared repository model The shared repository model, as GitHub aptly calls it, is when you push new branches directly to the source repository. From there, you can create a new pull request by comparing between branches, as we will see in the following sections. Of course, in order to be able to push to a repository you either have to be the owner or a collaborator; in other words you must have write access. Create branches in your fork – the fork and pull model Forked repositories are related to their parent in a way that GitHub uses in order to compare their branches. The fork and pull model is usually used in projects when one does not have write access but is willing to contribute. After forking a repository, you push a branch to your fork and then create a pull request in the source repository asking its maintainer to merge the changes. This is common practice to contribute to open source projects hosted on GitHub. You will not have access to their repository, but being open source, you can fork the public repository and work on your own copy. How to create and submit a pull request There are quite a few ways to initiate the creation of a pull request, as we you will see in the following sections. The most common one is to push a branch to your repository and let GitHub's UI guide you. Let's explore this option first. Use the Compare & pull request button Whenever a new branch is pushed to a repository, GitHub shows a quick button to create a pull request. In reality, you are taken to the compare page, as we will explore in the next section, but some values are already filled out for you. Let's create, for example, a new branch named add_gitignore where we will add a .gitignore file with the following contents: git checkout -b add_gitignore echo -e '.bundlen.sass-cachen.vendorn_site' > .gitignore git add .gitignore git commit -m 'Add .gitignore' git push origin add_gitignore Next, head over your repository's main page and you will notice the Compare & pull request button, as shown in the following screenshot: From here on, if you hit this button you will be taken to the compare page. Note that I am pushing to my repository following the shared repository model, so here is how GitHub greets me: What would happen if I used the fork and pull repository model? For this purpose, I created another user to fork my repository and followed the same instructions to add a new branch named add_gitignore with the same changes. From here on, when you push the branch to your fork, the Compare & pull request button appears whether you are on your fork's page or on the parent repository. Here is how it looks if you visit your fork: The following screenshot will appear, if you visit the parent repository: In the last case (captured in red), you can see from which user this branch came from (axil43:add_gitignore). In either case, when using the fork and pull model, hitting the Compare & pull request button will take you to the compare page with slightly different options: Since you are comparing across forks, there are more details. In particular, you can see the base fork and branch as well as the head fork and branch that are the ones you are the owner of. GitHub considers the default branch set in your repository to be the one you want to merge into (base) when the Create Pull Request button appears. Before submitting it, let's explore the other two options that you can use to create a pull request. You can jump to the Submit a pull request section if you like. Use the compare function directly As mentioned in the previous section, the Compare & pull request button gets you on the compare page with some predefined values. The button appears right after you push a new branch and is there only for a few moments. In this section, we will see how to use the compare function directly in order to create a pull request. You can access the compare function by clicking on the green button next to the branch drop-down list on a repository's main page: This is pretty powerful as one can compare across forks or, in the same repository, pretty much everything—branches, tags, single commits and time ranges. The default page when you land on the compare page is like the following one; you start by comparing your default branch with GitHub, proposing a list of recently created branches to choose from and compare: In order to have something to compare to, the base branch must be older than what you are comparing to. From here, if I choose the add_gitignore branch, GitHub compares it to a master and shows the diff along with the message that it is able to be merged into the base branch without any conflicts. Finally, you can create the pull request: Notice that I am using the compare function while I'm at my own repository. When comparing in a repository that is a fork of another, the compare function slightly changes and automatically includes more options as we have seen in the previous section. As you may have noticed the Compare & pull request quick button is just a shortcut for using compare manually. If you want to have more fine-grained control on the repositories and the branches compared, use the compare feature directly. Use the GitHub web editor So far, we have seen the two most well-known types of initiating a pull request. There is a third way as well: using entirely the web editor that GitHub provides. This can prove useful for people who are not too familiar with Git and the terminal, and can also be used by more advanced Git users who want to propose a quick change. As always, according to the model you are using (shared repository or fork and pull), the process is a little different. Let's first explore the shared repository model flow using the web editor, which means editing files in a repository that you own. The shared repository model Firstly, make sure you are on the branch that you wish to branch off; then, head over a file you wish to change and press the edit button with the pencil icon: Make the change you want in that file, add a proper commit message, and choose Create a new branch giving the name of the branch you wish to create. By default, the branch name is username-patch-i, where username is your username and i is an increasing integer starting from 1. Consecutive edits on files will create branches such as username-patch-1, username-patch-2, and so on. In our example, I decided to give the branch a name of my own: When ready, press the Propose file change button. From this moment on, the branch is created with the file edits you made. Even if you close the next page, your changes will not be lost. Let's skip the pull request submission for the time being and see how the fork and pull model works. The fork and pull model In the fork and pull model, you fork a repository and submit a pull request from the changes you make in your fork. In the case of using the web editor, there is a caveat. In order to get GitHub automatically recognize that you wish to perform a pull request in the parent repository, you have to start the web editor from the parent repository and not your fork. In the following screenshot, you can see what happens in this case: GitHub informs you that a new branch will be created in your repository (fork) with the new changes in order to submit a pull request. Hitting the Propose file change button will take you to the form to submit the pull request: Contrary to the shared repository model, you can now see the base/head repositories and branches that are compared. Also, notice that the default name for the new branch is patch-i, where i is an increasing integer number. In our case, this was the first branch created that way, so it was named patch-1. If you would like to have the ability to name the branch the way you like, you should follow the shared repository model instructions as explained in preceding section. Following that route, edit the file in your fork where you have write access, add your own branch name, hit the Propose file change button for the branch to be created, and then abort when asked to create the pull request. You can then use the Compare & pull request quick button or use the compare function directly to propose a pull request to the parent repository. One last thing to consider when using the web editor, is the limitation of editing one file at a time. If you wish to include more changes in the same branch that GitHub created for you when you first edited a file, you must first change to that branch and then make any subsequent changes. How to change the branch? Simply choose it from the drop-down menu as shown in the following screenshot: Submit a pull request So far, we have explored the various ways to initiate a pull request. In this section, we will finally continue to submit it as well. The pull request form is identical to the form when creating a new issue. If you have write access to the repository that you are making the pull request to, then you are able to set labels, milestone, and assignee. The title of the pull request is automatically filled by the last commit message that the branch has, or if there are multiple commits, it will just fill in the branch name. In either case, you can change it to your liking. In the following image, you can see the title is taken from the branch name after GitHub has stripped the special characters. In a sense, the title gets humanized: You can add an optional description and images if you deem proper. Whenever ready, hit the Create pull request button. In the following sections, we will explore how the peer review works. Peer review and inline comments The nice thing about pull requests is that you have a nice and clear view of what is about to get merged. You can see only the changes that matter, and the best part is that you can fire up a discussion concerning those changes. In the previous section, we submitted the pull request so that it can be reviewed and eventually get merged. Suppose that we are collaborating with a team and they chime in to discuss the changes. Let's first check the layout of a pull request. Summary In this article, we explored the GitHub workflow and the various ways to perform a pull request, as well as the many features GitHub provides to make that workflow even smoother. This is how the majority of open source projects work when there are dozens of contributors involved. Resources for Article: Further resources on this subject: Git Teaches – Great Tools Don't Make Great Craftsmen[article] Maintaining Your GitLab Instance[article] Configuration [article]
Read more
  • 0
  • 0
  • 33829
Modal Close icon
Modal Close icon