How-To Tutorials

In this post series, we will walk through how to write a universal (or isomorphic) JavaScript app. Part 1 covered what a universal JavaScript application is, why it is such an exciting concept, and the first two steps for creating our app, which are serving post data and adding React. In this second part of the series, we walk through steps 3-6, which are client-side routing with React Router, server rendering, data flow refactoring, and data loading of the app. Let’s get started. Save on some of our very best React and Angular product from the 7th to 13th November - it's a perfect opportunity to get stuck into two tools that are truly redefining modern web development. Save 50% on featured eBooks and 80% on featured video courses here. Step 3: Client-side routing with React Router git checkout client-side-routing && npm install Now that we're pulling and displaying posts, let's add some navigation to individual pages for each post. To do this, we will turn our list of posts from step 2 (see the Part 1 post) into links that are always present on the page. Each post will live at http://localhost:3000/:postId/:postSlug. We can use React Router and a routes.js file to set up this structure: // components/routes.js import React from 'react' import { Route } from 'react-router' import App from './App' import Post from './Post' module.exports = ( <Route path="/" component={App}> <Route path="/:postId/:postName" component={Post} /> </Route> ) We've changed the render method in App.js to render links to posts instead of just <li> tags: // components/App.js import React from 'react' import { Link } from 'react-router' const allPostsUrl = '/api/post' class App extends React.Component { constructor(props) { super(props) this.state = { posts: [] } } ... render() { const posts = this.state.posts.map((post) => { const linkTo = `/${post.id}/${post.slug}`; return ( <li key={post.id}> <Link to={linkTo}>{post.title}</Link> </li> ) }) return ( <div> <h3>Posts</h3> <ul> {posts} </ul> {this.props.children} </div> ) } } export default App And, we'll add a Post.js component to render each post's content: // components/Post.js import React from 'react' class Post extends React.Component { constructor(props) { super(props) this.state = { title: '', content: '' } } fetchPost(id) { const request = new XMLHttpRequest() request.open('GET', '/api/post/' + id, true) request.setRequestHeader('Content-type', 'application/json'); request.onload = () => { if (request.status === 200) { const response = JSON.parse(request.response) this.setState({ title: response.title, content: response.content }); } } request.send(); } componentDidMount() { this.fetchPost(this.props.params.postId) } componentWillReceiveProps(nextProps) { this.fetchPost(nextProps.params.postId) } render() { return ( <div> <h3>{this.state.title}</h3> <p>{this.state.content}</p> </div> ) } } export default Post The componentDidMount() and componentWillReceiveProps() methods are important because they let us know when we should fetch a post from the server. componentDidMount() will handle the first time the Post.js component is rendered, and then componentWillReceiveProps() will take over as React Router handles rerendering the component with different props. Run npm build:client && node server.js again to build and run the app. You will now be able to go to http://localhost:3000 and navigate around to the different posts. However, if you try to refresh on a single post page, you will get something like Cannot GET /3/debugging-node-apps. That's because our Express server doesn't know how to handle that kind of route. React Router is handling it completely on the front end. Onward to server rendering! Step 4: Server rendering git checkout server-rendering && npm install Okay, now we're finally getting to the good stuff. In this step, we'll use React Router to help our server take application requests and render the appropriate markup. To do that, we need to also build a server bundle like we build a client bundle, so that the server can understand JSX. Therefore, we've added the below webpack.server.config.js: // webpack.server.config.js var fs = require('fs') var path = require('path') module.exports = { entry: path.resolve(__dirname, 'server.js'), output: { filename: 'server.bundle.js' }, target: 'node', // keep node_module paths out of the bundle externals: fs.readdirSync(path.resolve(__dirname, 'node_modules')).concat([ 'react-dom/server', 'react/addons', ]).reduce(function (ext, mod) { ext[mod] = 'commonjs ' + mod return ext }, {}), node: { __filename: true, __dirname: true }, module: { loaders: [ { test: /.js$/, exclude: /node_modules/, loader: 'babel-loader?presets[]=es2015&presets[]=react' } ] } } We've also added the following code to server.js: // server.js import React from 'react' import { renderToString } from 'react-dom/server' import { match, RouterContext } from 'react-router' import routes from './components/routes' const app = express() ... app.get('*', (req, res) => { match({ routes: routes, location: req.url }, (err, redirect, props) => { if (err) { res.status(500).send(err.message) } else if (redirect) { res.redirect(redirect.pathname + redirect.search) } else if (props) { const appHtml = renderToString(<RouterContext {...props} />) res.send(renderPage(appHtml)) } else { res.status(404).send('Not Found') } }) }) function renderPage(appHtml) { return ` <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Universal Blog</title> </head> <body> <div id="app">${appHtml}</div> <script src="/bundle.js"></script> </body> </html> ` } ... Using React Router's match function, the server can find the appropriate requested route, renderToString, and send the markup down the wire. Run npm start to build the client and server bundles and start the app. Fantastic right? We're not done yet. Even though the markup is being generated on the server, we're still fetching all the data client side. Go ahead and click through the posts with your dev tools open, and you'll see the requests. It would be far better to load the data while we're rendering the markup instead of having to request it separately on the client. Since server rendering and universal apps are still bleeding-edge, there aren't really any established best practices for data loading. If you're using some kind of Flux implementation, there may be some specific guidance. But for this use case, we will simply grab all the posts and feed them through our app. In order to this, we first need to do some refactoring on our current architecture. Step 5: Data Flow Refactor git checkout data-flow-refactor && npm install It's a little weird how each post page has to make a request to the server for its content, even though the App component already has all the posts in its state. A better solution would be to have an App simply pass the appropriate content down to the Post component. // components/routes.js import React from 'react' import { Route } from 'react-router' import App from './App' import Post from './Post' module.exports = ( <Route path="/" component={App}> <Route path="/:postId/:postName" /> </Route> ) In our routes.js, we've made the Post route a componentless route. It's still a child of the App route, but now has to completely rely on the App component for rendering. Below are the changes to App.js: // components/App.js ... render() {    const posts = this.state.posts.map((post) => {      const linkTo = `/${post.id}/${post.slug}`;      return (        <li key={post.id}>          <Link to={linkTo}>{post.title}</Link>        </li>      )    })    const { postId, postName } = this.props.params;    let postTitle, postContent    if (postId && postName) {      const post = this.state.posts.find(p => p.id == postId)      postTitle = post.title      postContent = post.content    }    return (      <div>        <h3>Posts</h3>        <ul>          {posts}        </ul>        {postTitle && postContent ? (          <Post title={postTitle} content={postContent} />        ) : (          <h1>Welcome to the Universal Blog!</h1>        )}      </div>    ) } } export default App If we are on a post page, then props.params.postId and props.params.postName will both be defined and we can use them to grab the desired post and pass the data on to the Post component to be rendered. If those properties are not defined, then we're on the home page and can simply render a greeting. Now, our Post.js component can be a simple stateless functional component that simply renders its properties. // components/Post.js import React from 'react' const Post = ({title, content}) => ( <div> <h3>{title}</h3> <p>{content}</p> </div>) export default Post With that refactoring complete, we're ready to implement data loading. Step 6: Data Loading git checkout data-loading && npm install For this final step, we just need to make two small changes in server.js and App.js: // server.js ... app.get('*', (req, res) => { match({ routes: routes, location: req.url }, (err, redirect, props) => { if (err) { res.status(500).send(err.message) } else if (redirect) { res.redirect(redirect.pathname + redirect.search) } else if (props) { const routerContextWithData = ( <RouterContext {...props} createElement={(Component, props) => { return <Component posts={posts} {...props} /> }} /> ) const appHtml = renderToString(routerContextWithData) res.send(renderPage(appHtml)) } else { res.status(404).send('Not Found') } }) }) ... // components/App.js import React from 'react' import Post from './Post' import { Link, IndexLink } from 'react-router' const allPostsUrl = '/api/post' class App extends React.Component { constructor(props) { super(props) this.state = { posts: props.posts || [] } } ... In server.js, we're changing how the RouterContext creates elements by overwriting its createElement function and passing in our data as additional props. These props will get passed to any component that is matched by the route, which in this case will be our App component. Then, when the App component is initialized, it sets its posts state property to what it got from props or an empty array. That's it! Run npm start one last time, and cruise through your app. You can even disable JavaScript, and the app will automatically degrade to requesting whole pages. Thanks for reading! About the author John Oerter is a software engineer from Omaha, Nebraska, USA. He has a passion for continuous improvement and learning in all areas of software development, including Docker, JavaScript, and C#. He blogs at here.

  Microsoft Dynamics AX 2009 Administration A practical and efficient approach to planning, installing and configuring your Dynamics AX 2009 environment. Effectively consolidate and standardize processes across your organization with a centralized source for a variety of business needs Discover how to effectively plan and implement Dynamics AX 2009 in your business and fully grasp the necessary hardware, network, and software requirements to do so Understand the varied components that make up your Dynamics AX environment, including the role of base server components, how to implement the Enterprise Portal and Role Centers for web browser access to Dynamics AX, and the various workflow capabilities required to fit the needs of your company Get to grips with Dynamics AX data components and understand when and how to exchange data synchronously or asynchronously and migrate data Includes carefully organized instructions along with screenshots that clearly display how to administer and configure your Dynamics AX environment with ease Workflow prerequisites Before we begin installing and setting up Workflow, you will need to have administrator privileges on the machines in which you are installing Workflow on. The following prerequisites are required: Internet Information Services (IIS) 7 .NET Framework 2.0 Business Connector The Workflow component for Dynamics AX utilizes the Business Connector to communicate directly to Dynamics AX from its web service. Although it depends on the number of users and computing resources available, it is best practice to implement the Workflow web service application on its own server. Workflow accounts setup The Workflow system in Dynamics AX utilizes two accounts to function properly. If these accounts are not specified, workflow will still function; however, it is best practice to have dedicated accounts. One account is the service account. This account is responsible for the communication between Dynamics AX and the Workflow web service. The other account is the execution account. This account is responsible for executing Workflow tasks and processes. Similar to the Business Connector proxy account, these two accounts must be created with the following criteria: The password must never expire It must not be interactive It must not be associated to any Dynamics AX users Once the Workflow accounts have been created, the next process is to ensure that the Dynamics AX Workflow system will use the accounts. To do this, perform the following steps: Log in to Dynamics AX 2009. Go to Administration | Setup | Security | System service accounts. In the System service accounts form, specify the Workflow System Account and the Workflow Execution Account. To specify the accounts that were created in the Active Directory, mark the Alias field radio boxes. When complete, click on the OK button. Now that the Workflow accounts have been specified, the Dynamics AX Workflow system can utilize these accounts when communicating with the Workflow web service. It is also possible to create a user within Dynamics AX but not Active Directory, and use the Dynamics AX user accounts as the Workflow Service and Execution accounts. These accounts may appear as different users in Dynamics AX; however, these accounts will be impersonated by the AOS service account to the Workflow web service. This can make troubleshooting and connection auditing more difficult and therefore, it is not recommended. Installing Workflow Since Workflow consists of various parts that function together to create the Dynamics AX 2009 Workflow system, we will break down each part’s setup and complete each setup individually. Creating a website for Workflow Before we can install the Workflow extended server component, we must have a website available to install upon. It is possible to use the default website that is on port 80. However, it is not recommended; therefore, a new website must be created. For information regarding how to create a website in IIS 7, refer to http://technet.microsoft.com/en-us/library/cc772350(WS.10).aspx. Installing the Workflow component By now you should be accustomed to the process of installing the extended components for Dynamics AX. Installing Workflow is no different. The following steps will guide you through the process: Run the Microsoft Dynamics AX Setup wizard to add new components. In the Add or modify components screen of the wizard, mark the Workflow checkbox, as shown in the following screenshot and then click on the Next button: In the following screen of the wizard, specify the password for the .NET Business Connector proxy account, and then click on the Next button: The Domainuser name field will automatically be populated if the Business Connector proxy user is specified in the Administration | Setup | Security | System service accounts form in the Business Connector Proxy group. In the next section of the wizard, you will be prompted to tweak the Workflow service. In other words, you can select which website you want to install the Workflow service into. By default, the wizard will select the default website in IIS. However, it is recommended to install Workflow on its own dedicated site and port, you have the flexibility to do so. When you are satisfied with the settings on this page, click on the Next button. In the following screen you will be prompted to specify the AOS account for the Workflow service to grant permissions to. If there is more than one AOS and each AOS service account is different, provide the accounts for each AOS so that they can access the Workflow service. In the following step, you are prompted to complete the installation of the Workflow by clicking on the Finish button. You will want to restart IIS after the Workflow has been successfully installed. Therefore, leave the option checked at the bottom, as shown in the following screenshot: Once installed, you will be prompted with the final screen. The final screen will display the result of the installation of the Dynamics AX 2009 Workflow system. If the installation is successful, you will see a green box next to the installed component. Otherwise, if the box is orange or red, you should open the log file after you close the wizard by marking the checkbox at the bottom. Enabling Workflow in Windows Server 2008 R2 In Windows Server 2008 R2, additional setup is required to enable the Workflow web service. The Workflow web service application pool must be enabled to run 32-bit applications. Otherwise, the Workflow service will fail. To set up the application pool to run 32-bit applications, perform the following steps: Go to the IIS console and access the web server that you installed the Workflow web service on. Under &ltYour web server&gt | Application Pools, select the Dynamics AX Workflow application pool. Under Actions, go to Advanced Settings.... In the Advanced Settings window, set the Enable 32-Bit Applications property to True. When complete, click on the OK button to save the changes. After performing these steps, the Workflow web service can then be used by Dynamics AX to process Workflows. Next, we will set up Dynamics AX to use the Workflow web service.

In this article, Marco Schwartz and Stefan Buttigieg, the authors of the Arduino Android Blueprints, we are going to use most of the concepts we have learned to control a mobile robot via an Android app. The robot will have two motors that we can control, and also an ultrasonic sensor in the front so that it can detect obstacles. The robot will also have a BLE chip so that it can receive commands from the Android app. The following will be the major takeaways: Building a mobile robot based on the Arduino platform Connecting a BLE module to the Arduino robot (For more resources related to this topic, see here.) Configuring the hardware We are first going to assemble the robot itself, and then see how to connect the Bluetooth module and the ultrasonic sensor. To give you an idea of what you should end up with, the following is a front-view image of the robot when fully assembled: The following image shows the back of the robot when fully assembled: The first step is to assemble the robot chassis. To do so, you can watch the DFRobot assembly guide at https://www.youtube.com/watch?v=tKakeyL_8Fg. Then, you need to attach the different Arduino boards and shields to the robot. Use the spacers found in the robot chassis kit to mount the Arduino Uno board first. Then put the Arduino motor shield on top of that. At this point, use the screw header terminals to connect the two DC motors to the motor shield. This is how it should look at this point: Finally, mount the prototyping shield on top of the motor shield. We are now going to connect the BLE module and the ultrasonic sensor to the Arduino prototyping shield. The following is a schematic diagram showing the connections between the Arduino Uno board (done via the prototyping shield in our case) and the components: Now perform the following steps: First, we are now going to connect the BLE module. Place the module on the prototyping shield. Connect the power supply of the module as follows: GND goes to the prototyping shield's GND pin, and VIN goes to the prototyping shield's +5V. After that, you need to connect the different wires responsible for the SPI interface: SCK to Arduino pin 13, MISO to Arduino pin 12, and MOSI to Arduino pin 11. Then connect the REQ pin to Arduino pin 10. Finally, connect the RDY pin to Arduino pin 2 and the RST pin to Arduino pin 9. For the URM37 module, connect the VCC pin of the module to Arduino +5V, GND to GND, and the PWM pin to the Arduino A3 pin. To review the pin order on the URM37 module, you can check the official DFRobot documentation at http://www.dfrobot.com/wiki/index.php?title=URM37_V3.2_Ultrasonic_Sensor_(SKU:SEN0001). The following is a close-up image of the prototyping shield with the BLE module connected: Finally, connect the 7.4 V battery to the Arduino Uno board power jack. The battery is simply placed below the Arduino Uno board. Testing the robot We are now going to write a sketch to test the different functionalities of the robot, first without using Bluetooth. As the sketch is quite long, we will look at the code piece by piece. Before you proceed, make sure that the battery is always plugged into the robot. Now perform the following steps: The sketch starts by including the aREST library that we will use to control the robot via serial commands: #includ e <aREST.h> Now we declare which pins the motors are connected to: int speed_motor1 = 6; int speed_motor2 = 5; int direction_motor1 = 7; int direction_motor2 = 4; We also declare which pin the ultrasonic sensor is connected to: int distance_sensor = A3; Then, we create an instance of the aREST library: aREST rest = aREST(); To store the distance data measured by the ultrasonic sensor, we declare a distance variable: int distance; In the setup() function of the sketch, we first initialize serial communications that we will use to communicate with the robot for this test: Serial.begin(115200); We also expose the distance variable to the REST API, so we can access it easily: rest.variable("distance",&distance); To control the robot, we are going to declare a whole set of functions that will perform the basic operations: going forward, going backward, turning on itself (left or right), and stopping. We will see the details of these functions in a moment; for now, we just need to expose them to the API: rest.function("forward",forward); rest.function("backward",backward); rest.function("left",left); rest.function("right",right); rest.function("stop",stop); We also give the robot an ID and a name: rest.set_id("001"); rest.set_name("mobile_robot"); In the loop() function of the sketch, we first measure the distance from the sensor: distance = measure_distance(distance_sensor); We then handle the requests using the aREST library: rest.handle(Serial); Now, we will look at the functions for controlling the motors. They are all based on a function to control a single motor, where we need to set the motor pins, the speed, and the direction of the motor: void send_motor_command(int speed_pin, int direction_pin, int pwm, boolean dir) { analogWrite(speed_pin, pwm); // Set PWM control, 0 for stop, and 255 for maximum speed digitalWrite(direction_pin, dir); // Dir set the rotation direction of the motor (true or false means forward or reverse) } Based on this function, we can now define the different functions to move the robot, such as forward: int forward(String command) { send_motor_command(speed_motor1,direction_motor1,100,1); send_motor_command(speed_motor2,direction_motor2,100,1); return 1; } We also define a backward function, simply inverting the direction of both motors: int backward(String command) { send_motor_command(speed_motor1,direction_motor1,100,0); send_motor_command(speed_motor2,direction_motor2,100,0); return 1; } To make the robot turn left, we simply make the motors rotate in opposite directions: int left(String command) { send_motor_command(speed_motor1,direction_motor1,75,0); send_motor_command(speed_motor2,direction_motor2,75,1); return 1; } We also have a function to stop the robot: int stop(String command) { send_motor_command(speed_motor1,direction_motor1,0,1); send_motor_command(speed_motor2,direction_motor2,0,1); return 1; } There is also a function to make the robot turn right, which is not detailed here. We are now going to test the robot. Before you do anything, ensure that the battery is always plugged into the robot. This will ensure that the motors are not trying to get power from your computer USB port, which could damage it. Also place some small support at the bottom of the robot so that the wheels don't touch the ground. This will ensure that you can test all the commands of the robot without the robot moving too far from your computer, as it is still attached via the USB cable. Now you can upload the sketch to your Arduino Uno board. Open the serial monitor and type the following: /forward This should make both the wheels of the robot turn in the same direction. You can also try the other commands to move the robot to make sure they all work properly. Then, test the ultrasonic distance sensor by typing the following: /distance You should get back the distance (in centimeters) in front of the sensor: {"distance": 24, "id": "001", "name": "mobile_robot", "connected": true} Try changing the distance by putting your hand in front of the sensor and typing the command again. Writing the Arduino sketch Now that we have made sure that the robot is working properly, we can write the final sketch that will receive the commands via Bluetooth. As the sketch shares many similarities with the test sketch, we are only going to see what is added compared to the test sketch. We first need to include more libraries: #include <SPI.h> #include "Adafruit_BLE_UART.h" #include <aREST.h> We also define which pins the BLE module is connected to: #define ADAFRUITBLE_REQ 10 #define ADAFRUITBLE_RDY 2     // This should be an interrupt pin, on Uno thats #2 or #3 #define ADAFRUITBLE_RST 9 We have to create an instance of the BLE module: Adafruit_BLE_UART BTLEserial = Adafruit_BLE_UART(ADAFRUITBLE_REQ, ADAFRUITBLE_RDY, ADAFRUITBLE_RST); In the setup() function of the sketch, we initialize the BLE chip: BTLEserial.begin(); In the loop() function, we check the status of the BLE chip and store it in a variable: BTLEserial.pollACI(); aci_evt_opcode_t status = BTLEserial.getState(); If we detect that a device is connected to the chip, we handle the incoming request with the aREST library, which will allow us to use the same commands as before to control the robot: if (status == ACI_EVT_CONNECTED) { rest.handle(BTLEserial); } You can now upload the code to your Arduino board, again by making sure that the battery is connected to the Arduino Uno board via the power jack. You can now move on to the development of the Android application to control the robot. Summary In this article, we managed to create our very own mobile robot together with a companion Android application that we can use to control our robot. We achieved this step by step by setting up an Arduino-enabled robot and coding the companion Android application. It uses the BLE software and hardware of an Android physical device running on Android 4.3 or higher. Resources for Article: Further resources on this subject: Our First Project – A Basic Thermometer [article] Hardware configuration [article] Avoiding Obstacles Using Sensors [article]

In this article, Riaz Ahmed, the author of the book Learning SAP Analytics Cloud provides an overview of this unique cloud-based business intelligence platform. You will learn about the following SAP Analytics Cloud segments Models and data sources Visualization Collaboration Presentation Administration and security (For more resources related to this topic, see here.) What is SAP Analytics Cloud? SAP Analytics Cloud is a new generation cloud-based application, which helps you explore your data, perform visualization and analysis, create financial plans, and produce predictive forecasting. It is a one-stop-shop solution to cope with your analytic needs and comprises business intelligence, planning, predictive analytics, and governance and risk. The application is built on the SAP Cloud Platform and delivers great performance and scalability. In addition to the on-premise SAP HANA, SAP BW, and S/4HANA sources, you can work with data from a wide range of non-SAP sources including Google Drive, Salesforce, SQL Server, Concur, and CSV to name a few. SAP Analytics Cloud allows you to make secure connections to these cloud and on premise data sources. Anatomy of SAP Analytics Cloud The following figure depicts the anatomy of SAP Analytics Cloud: Data sources and models Before commencing your analytical tasks in SAP Analytics Cloud, you need to create  models. Models are the basis for all of  your analysis in SAP Analytics Cloud  to evaluate the performance of your organization. It is a high-level design that exposes the analytic requirements of end users. You can create planning and analytics models based on the cloud or on premise data sources. Analytics models are more simpler and flexible, while planning models are full featured models in which business analysts and finance professionals can quickly and easily build connected models to analyze data and then collaborate with each other to attain better business performance. Preconfigured with dimensions for Time and Categories, planning models support for multi-currency and security features at both model and dimension levels. After creating these models you can share it with other users in your organization. Before sharing, you can set up model access privileges for users according to their level of authorization and can also enable data auditing. With the help of SAP Analytics Cloud's analytical capabilities, users can discover hidden traits in the data and can predict likely outcomes. It equips them with the ability to uncover potentials risks and hidden opportunities. To determine what content to include in your model, you must first identify the columns from the source data on which users need to query. The columns you need in your model reside in some sort of data source. SAP Analytics Cloud supports three types of data sources: files (such as CSV or Excel files) that usually reside on your computer, live data connection from a connected remote system, and  cloud apps. In addition to the files on your computer, you can use on-premise data sources such as SAP Business Warehouse, SAP ERP, SAP Universe, SQL database, and more to acquire data for your models. In the cloud, you can get data from apps like Concur, Google Drive, SAP Business ByDesign, SAP Hybris Cloud, OData Services, and Success Factors. The following figure depicts these data sources. The cloud apps data sources you can use with SAP Analytics Cloud are displayed above the firewall mark, while those in your local network are shown under the firewall.As you can see in this figure, there are over twenty data sources currently supported by SAP Analytics Cloud.  The method of connecting to these data sources also vary from each other. Create a direct live connection to SAP HANA You can connect to on-premise SAP HANA system to use live data in SAP Analytics Cloud. Live data means that you can get up-to-the-minute data when you open a story in SAP Analytics Cloud.In this case, any changes made to the data in the source system are reflected immediately. Usually, there are two ways to establish a connection to a data source – use  the Connection option from the main menu, or specify the data source during the process of creating a model. However, live data connections must be established via the Connection menu option prior to creating the corresponding model. Connect remote systems to import data In addition to creating live connections, you can also create connections that allow you to import data into SAP Analytics Cloud. In these types of connections that you make to access remote systems, data is imported (copied) to SAP Analytics Cloud. Any changes users make in the source data do not affect the imported data. To establish connections with these remote systems, you need to install some additional components. For example, you must install SAP HANA Cloud Connector to access SAP Business Planning and Consolidation (BPC) for Netweaver. Similarly, SAP Analytics Cloud Agent should be installed for SAP Business Warehouse (BW), SQL Server, SAP ERP, and others. Connect to a cloud app to import data In addition to creating a live connection to create a model on live data and importing data from remote systems, you can set up connections to acquire data from cloud apps, such as Google Drive, SuccessFactors, Odata, Concur, Fieldglass, Google BigQuery, and more. Refreshing imported data SAP Analytics Cloud allows you to refresh your imported data. With this option, you can reimport the data on demand to get the latest values. You can perform this refresh operation either manually, or create an import schedule to refresh the data at a specific date and time or on a recurring basis. Visualization Once you have created a model and set up appropriate security for it, you can create stories in which the underlying model data can be explored and visualized with the help of different types of charts, geo maps, and tables. There is a wide range of charts you can add to your story pages to address different scenarios. You can create multiple pages in your story to present your model data using charts, geo maps, tables, text, shapes, and images. On your story pages, you can link dimensions to present data from multiple sources. Adding reference lines and thresholds, applying filters, and drilling down into data can be done on-the-fly. The ability to interactively drag and drop page objects is the most useful feature of this application. Charts: SAP Analytics Cloud comes with a variety of charts to present your analysis according to your specific needs. You can add multiple types of charts to a single story page. Geo Map: The models that include latitude and longitude information can be used in stories to visualize data in geo maps. By adding multiple layers of different types of data in geo maps, you can show different geographic features and point of interests enabling you to perform sophisticated geographic analysis. Table: A table is a spreadsheet like object that can be used to view and analyze text data. You can add this object to either canvas or grid pages in stories. Static and Dynamic Text: You can add static and dynamic text to your story pages. Static text are normally used to display page titles, while dynamic text automatically updates page headings based on the values from the source input control or filter. Images and Shapes: You can add images (such as your company logo) to your story page by uploading them from your computer. In addition to images, you can also add shapes such as line, square, or circle to your page. Collaboration Collaboration, alert, and notification features of SAP Analytics Cloud keep business users in touch with each other while executing their tasks.During the process of creating models and stories, you need input from other people. For example, you might ask a colleague to update a model by providing first quarter sales data, or request Sheela to enter her comments on one of your story pages. In SAP Analytics Cloud, these interactions come under the collaboration features of the application. Using these features users can discuss business content and share information that consequently smoothens the decision making process. Here is a list of available collaboration features in the application that allow group members to discuss stories and other business content. Create a workflow using events and tasks: The events and tasks features in SAP Analytics Cloud are the two major sources that help you collaborate with other group members and  manage your planning and analytic activities. After creating an event and assigning tasks to relevant group members, you can monitor the task progress in the Events interface. Here is the workflow to utilize these two features: Create events based on categories and processes within categories Create a task, assign it to users, and set a due date for its submission Monitor the task progress Commenting on a chart's data point: Using this feature, you can add annotations or additional information to individual data points in a chart. Commenting on a story page: In addition to adding comments to an individual chart, you have the option to add comments on an entire story page to provide some vital information to other users. When you add a comment, other users can see and reply to it. Produce a story as a PDF: You can save your story in a PDF file to share it with other users and for offline access. You can save all story pages or a specific page as a PDF file. Sharing a story with colleagues: Once you complete a story, you can share it with other members in your organization. You are provided with three options (Public, Teams, and Private) when you save a story. When you save your story in the Public folder, it can be accessed by anyone. The Teams option lets you select specific teams to share your story with. In the Private option, you have to manually select users with whom you want to share your story. Collaborate via discussions: You can collaborate with colleagues using the discussions feature of SAP Analytics Cloud. The discussions feature enables you to connect with other members in real-time. Sharing files and other objects: The Files option under the Browse menu allows you to access SAP Analytics Cloud repository, where the stories you created and the files you uploaded are stored. After accessing the Files page, you can share its objects with other users. On the Files page, you can manage files and folders, upload files, share files, and change share settings. Presentation In the past, board meetings were held in which the participants used to deliver their reports through their laptops and a bunch of papers. With different versions of reality, it was very difficult for decision makers to arrive at a good decision. With the advent of SAP Digital Boardroom the board meetings have revolutionized. It is a next-generation presentation platform, which helps you visualize your data and plan ahead in a real-time environment. It runs on multiple screens simultaneously displaying information from difference sources. Due to this capability, more people can work together using a single dataset that consequently creates one version of reality to make the best decision. SAP Digital Boardroom is changing the landscape of board meetings. In addition to supporting traditional presentation methods, it goes beyond the corporate boardroom and allows remote members to join and participate the meeting online. These remote participants can actively engage in the meeting and can play with live data using their own devices. SAP Digital Boardroom is a visualization and presentation platform that enormously assists in the decision-making process. It transforms executive meetings by replacing static and stale presentations with interactive discussions based on live data, which allows them to make fact-based decisions to drive their business. Here are the main benefits of SAP Digital Boardroom: Collaborate with others in remote locations and on other devices in an interactive meeting room. Answer ad hoc questions on the fly. Visualize, recognize, experiment, and decide by jumping on and off script at any point. Find answers to the questions that matter to you by exploring directly on live data and focusing on relevant aspects by drilling into details. Discover opportunities or reveal hidden threats. Simulate various decisions and project the results. Weigh and share the pros and cons of your findings. The Digital Boardroom is interactive so you can retrieve real-time data, make changes to the schedule and even run through what-if scenarios. It presents a live picture of your organization across three interlinked touch screens to make faster, better executive decisions. There are two aspects of Digital Boardroom with which you can share existing stories with executives and decision-makers to reveal business performance. First, you have to design your agenda for your boardroom presentation by adding meeting information, agenda items, and linking stories as pages in a navigation structure. Once you have created a digital boardroom agenda, you can schedule a meeting to discuss the agenda. In Digital Boardroom interface, you can organize a meeting in which members can interact with live data during a boardroom presentation. Administration andsecurity An application that is accessed by multiple users is useless without a proper system administration module. The existence of this module ensures that things are under control and secured. It also includes upkeep, configuration, and reliable operation of the application. This module is usually assigned to a person who is called system administrator and whose responsibility is to watch uptime, performance, resources, and security of the application. Being a multi-user application, SAP Analytics Cloud also comes with this vital module that allows a system administrator to take care of the following segments: Creating users and setting their passwords Importing users from another data source Exporting user's profiles for other apps Deleting unwanted user accounts from the system Creating roles and assigning them to users Setting permissions for roles Forming teams Setting security for models Monitoring users activities Monitoring data changes Monitoring system performance System deployment via export and import Signing up for trial version If you want to get your feet wet by exploring this exciting cloud application, then sign up for a free 30 days trial version. Note that the trial version doesn't allow you to access all the features of SAP Analytics Cloud. For example, you cannot create a planning model in the trial version nor can you access its security and administration features. Execute the following steps to get access to the free SAP Analytics Cloud trial: Put the following URL in you browser's address bar and press enter: http://discover.sapanalytics.cloud/trialrequest-auto/ Enter and confirm your business e-mail address in relevant boxes. Select No from the Is your company an SAP Partner? list. Click the Submit button. After a short while, you will get an e-mail with a link to connect to SAP Analytics Cloud. Click the Activate Account button in the e-mail. This will open the Activate Your Account page, where you will have to set a strong password. The password must be at least 8 characters long and should also include uppercase and lowercase letters, numbers, and symbols. After entering and confirming your password, click the Save button to complete the activation process. The confirmation page appears telling you that your account is successfully activated. Click Continue. You will be taken to SAP Analytics Cloud site. The e-mail you receive carries a link under SAP Analytics Cloud System section that you can use to access the application any time. Your username (your e-mail address) is also mentioned in the same e-mail along with a log on button to access the application. Summary SAP Analytics Cloud is the next generation cloud-based analytic application, which provides an end-to-end cloud analytics experience. SAP Analytics Cloud can help transform how you discover, plan, predict, collaborate, visualize, and extend all in one solution. In addition to on-premise data sources, you can fetch data from a variety of other cloud apps and even from Excel and text files to build your data models and then create stories based on these models. The ultimate purpose of this amazing and easy-to-use application is to enable you to make the right decision. SAP Analytics Cloud is more than visualization of data it is insight to action, it is realization of success. Resources for Article: Further resources on this subject: Working with User Defined Values in SAP Business One [article] Understanding Text Search and Hierarchies in SAP HANA [article] Meeting SAP Lumira [article]

In this article, by the author, Enrique Amodeo, of the book, Learning Behavior-driven Development with JavaScript, we will look into an advanced concept: how to test a user interface. For this purpose, you will learn the following topics: Using WebDriverJS to manipulate a browser and inspect the resulting HTML generated by our UI Organizing our UI codebase to make it easily testable The right abstraction level for our UI tests (For more resources related to this topic, see here.) Our strategy for UI testing There are two traditional strategies towards approaching the problem of UI testing: record-and-replay tools and end-to-end testing. The first approach, record-and-replay, leverages the use of tools capable of recording user activity in the UI and saves this into a script file. This script file can be later executed to perform exactly the same UI manipulation as the user performed and to check whether the results are exactly the same. This approach is not very compatible with BDD because of the following reasons: We cannot test-first our UI. To be able to use the UI and record the user activity, we first need to have most of the code of our application in place. This is not a problem in the waterfall approach, where QA and testing are performed after the codification phase is finished. However, in BDD, we aim to document the product features as automated tests, so we should write the tests before or during the coding. The resulting test scripts are low-level and totally disconnected from the problem domain. There is no way to use them as a live documentation for the requirements of the system. The resulting test suite is brittle and it will stop working whenever we make slight changes, even cosmetic ones, to the UI. The problem is that the tools record the low-level interaction with the system that depends on technical details of the HTML. The other classic approach is end-to-end testing, where we do not only test the UI layer, but also most of the system or even the whole of it. To perform the setup of the tests, the most common approach is to substitute the third-party systems with test doubles. Normally, the database is under the control of the development team, so some practitioners use a regular database for the setup. However, we could use an in-memory database or even mock the DAOs. In any case, this approach prompts us to create an integrated test suite where we are not only testing the correctness of the UI, but the business logic as well. In the context of this discussion, an integrated test is a test that checks several layers of abstraction, or subsystems, in combination. Do not confuse it with the act of testing several classes or functions together. This approach is not inherently against BDD; for example, we could use Cucumber.js to capture the features of the system and implement Gherkin steps using WebDriver to drive the UI and make assertions. In fact, for most people, when you say BDD they always interpret this term to refer to this kind of test. We will end up writing a lot of test cases, because we need to combine the scenarios from the business logic domain with the ones from the UI domain. Furthermore, in which language should we formulate the tests? If we use the UI language, maybe it will be too low-level to easily describe business concepts. If we use the business domain language, maybe we will not be able to test the important details of the UI because they are too low-level. Alternatively, we can even end up with tests that mix UI language with business terminology, so they will neither be focused nor very clear to anyone. Choosing the right tests for the UI If we want to test whether the UI works, why should we test the business rules? After all, this is already tested in the BDD test suite of the business logic layer. To decide which tests to write, we should first determine the responsibilities of the UI layer, which are as follows: Presenting the information provided by the business layer to the user in a nice way. Transforming user interaction into requests for the business layer. Controlling the changes in the appearance of the UI components, which includes things such as enabling/disabling controls, highlighting entry fields, showing/hiding UI elements, and so on. Orchestration between the UI components. Transferring and adapting information between the UI components and navigation between pages fall under this category. We do not need to write tests about business rules, and we should not assume much about the business layer itself, apart from a loose contract. How we should word our tests? We should use a UI-related language when we talk about what the user sees and does. Words such as fields, buttons, forms, links, click, hover, highlight, enable/disable, or show and hide are relevant in this context. However, we should not go too far; otherwise, our tests will be too brittle. Saying, for example, that the name field should have a pink border is too low-level. The moment that the designer decides to use red instead of pink, or changes his mind and decides to change the background color instead of the border, our test will break. We should aim for tests that express the real intention of the user interface; for example, the name field should be highlighted as incorrect. The testing architecture At this point, we could write tests relevant for our UI using the following testing architecture: A simple testing architecture for our UI We can use WebDriver to issue user gestures to interact with the browser. These user gestures are transformed by the browser in to DOM events that are the inputs of our UI logic and will trigger operations on it. We can use WebDriver again to read the resulting HTML in the assertions. We can simply use a test double to impersonate our server, so we can set up our tests easily. This architecture is very simple and sounds like a good plan, but it is not! There are three main problems here: UI testing is very slow. Take into account that the boot time and shutdown phase can take 3 seconds in a normal laptop. Each UI interaction using WebDriver can take between 50 and 100 milliseconds, and the latency with the fake server can be an extra 10 milliseconds. This gives us only around 10 tests per second, plus an extra 3 seconds. UI tests are complex and difficult to diagnose when they fail. What is failing? Our selectors used to tell WebDriver how to find the relevant elements. Some race condition we were not aware of? A cross-browser issue? Also note that our test is now distributed between two different processes, a fact that always makes debugging more difficult. UI tests are inherently brittle. We can try to make them less brittle with best practices, but even then a change in the structure of the HTML code will sometimes break our tests. This is a bad thing because the UI often changes more frequently than the business layer. As UI testing is very risky and expensive, we should try to code as less amount of tests that interact with the UI as possible. We can achieve this without losing testing power, with the following testing architecture:   A smarter testing architecture We have now split our UI layer into two components: the view and the UI logic. This design aligns with the family of MV* design patterns. In the context of this article, the view corresponds with a passive view, and the UI logic corresponds with the controller or the presenter, in combination with the model. A passive view is usually very hard to test; so in this article we will focus mostly on how to do it. You will often be able to easily separate the passive view from the UI logic, especially if you are using an MV* pattern, such as MVC, MVP, or MVVM. Most of our tests will be for the UI logic. This is the component that implements the client-side validation, orchestration of UI components, navigation, and so on. It is the UI logic component that has all the rules about how the user can interact with the UI, and hence it needs to maintain some kind of internal state. The UI logic component can be tested completely in memory using standard techniques. We can simply mock the XMLHttpRequest object, or the corresponding object in the framework we are using, and test everything in memory using a single Node.js process. No interaction with the browser and the HTML is needed, so these tests will be blazingly fast and robust. Then we need to test the view. This is a very thin component with only two responsibilities: Manipulating and updating the HTML to present the user with the information whenever it is instructed to do so by the UI logic component Listening for HTML events and transforming them into suitable requests for the UI logic component The view should not have more responsibilities, and it is a stateless component. It simply does not need to store the internal state, because it only transforms and transmits information between the HTML and the UI logic. Since it is the only component that interacts with the HTML, it is the only one that needs to be tested using WebDriver. The point of all of this is that the view can be tested with only a bunch of tests that are conceptually simple. Hence, we minimize the number and complexity of the tests that need to interact with the UI. WebDriverJS Testing the passive view layer is a technical challenge. We not only need to find a way for our test to inject native events into the browser to simulate user interaction, but we also need to be able to inspect the DOM elements and inject and execute scripts. This was very challenging to do approximately 5 years ago. In fact, it was considered complex and expensive, and some practitioners recommended not to test the passive view. After all, this layer is very thin and mostly contains the bindings of the UI to the HTML DOM, so the risk of error is not supposed to be high, specially if we use modern cross-browser frameworks to implement this layer. Nonetheless, nowadays the technology has evolved, and we can do this kind of testing without much fuss if we use the right tools. One of these tools is Selenium 2.0 (also known as WebDriver) and its library for JavaScript, which is WebDriverJS (https://code.google.com/p/selenium/wiki/WebDriverJs).  In this book, we will use WebDriverJS, but there are other bindings in JavaScript for Selenium 2.0, such as WebDriverIO (http://webdriver.io/). You can use the one you like most or even try both. The point is that the techniques I will show you here can be applied with any client of WebDriver or even with other tools that are not WebDriver. Selenium 2.0 is a tool that allows us to make direct calls to a browser automation API. This way, we can simulate native events, we can access the DOM, and we can control the browser. Each browser provides a different API and has its own quirks, but Selenium 2.0 will offer us a unified API called the WebDriver API. This allows us to interact with different browsers without changing the code of our tests. As we are accessing the browser directly, we do not need a special server, unless we want to control browsers that are on a different machine. Actually, this is only true, due some technical limitations, if we want to test against a Google Chrome or a Firefox browser using WebDriverJS. So, basically, the testing architecture for our passive view looks like this: Testing with WebDriverJS We can see that we use WebDriverJS for the following: Sending native events to manipulate the UI, as if we were the user, during the action phase of our tests Inspecting the HTML during the assert phase of our test Sending small scripts to set up the test doubles, check them, and invoke the update method of our passive view Apart from this, we need some extra infrastructure, such as a web server that serves our test HTML page and the components we want to test. As is evident from the diagram, the commands of WebDriverJS require some network traffic to able to send the appropriate request to the browser automation API, wait for the browser to execute, and get the result back through the network. This forces the API of WebDriverJS to be asynchronous in order to not block unnecessarily. That is why WebDriverJS has an API designed around promises. Most of the methods will return a promise or an object whose methods return promises. This plays perfectly well with Mocha and Chai.  There is a W3C specification for the WebDriver API. If you want to have a look, just visit https://dvcs.w3.org/hg/webdriver/raw-file/default/webdriver-spec.html. The API of WebDriverJS is a bit complex, and you can find its official documentation at http://selenium.googlecode.com/git/docs/api/javascript/module_selenium-webdriver.html. However, to follow this article, you do not need to read it, since I will now show you the most important API that WebDriverJS offers us. Finding and interacting with elements It is very easy to find an HTML element using WebDriverJS; we just need to use either the findElement or the findElements methods. Both methods receive a locator object specifying which element or elements to find. The first method will return the first element it finds, or simply fail with an exception, if there are no elements matching the locator. The findElements method will return a promise for an array with all the matching elements. If there are no matching elements, the promised array will be empty and no error will be thrown. How do we specify which elements we want to find? To do so, we need to use a locator object as a parameter. For example, if we would like to find the element whose identifier is order_item1, then we could use the following code: var By = require('selenium-webdriver').By;   driver.findElement(By.id('order_item1')); We need to import the selenium-webdriver module and capture its locator factory object. By convention, we store this locator factory in a variable called By. Later, we will see how we can get a WebDriverJS instance. This code is very expressive, but a bit verbose. There is another version of this: driver.findElement({ id: 'order_item1' }); Here, the locator criteria is passed in the form of a plain JSON object. There is no need to use the By object or any factory. Which version is better? Neither. You just use the one you like most. In this article, the plain JSON locator will be used. The following are the criteria for finding elements: Using the tag name, for example, to locate all the <li> elements in the document: driver.findElements(By.tagName('li'));driver.findElements({ tagName: 'li' }); We can also locate using the name attribute. It can be handy to locate the input fields. The following code will locate the first element named password: driver.findElement(By.name('password')); driver.findElement({ name: 'password' }); Using the class name; for example, the following code will locate the first element that contains a class called item: driver.findElement(By.className('item')); driver.findElement({ className: 'item' }); We can use any CSS selector that our target browser understands. If the target browser does not understand the selector, it will throw an exception; for example, to find the second item of an order (assuming there is only one order on the page): driver.findElement(By.css('.order .item:nth-of-type(2)')); driver.findElement({ css: '.order .item:nth-of-type(2)' }); Using only the CSS selector you can locate any element, and it is the one I recommend. The other ones can be very handy in specific situations. There are more ways of locating elements, such as linkText, partialLinkText, or xpath, but I seldom use them. Locating elements by their text, such as in linkText or partialLinkText, is brittle because small changes in the wording of the text can break the tests. Also, locating by xpath is not as useful in HTML as using a CSS selector. Obviously, it can be used if the UI is defined as an XML document, but this is very rare nowadays. In both methods, findElement and findElements, the resulting HTML elements are wrapped as a WebElement object. This object allows us to send an event to that element or inspect its contents. Some of its methods that allow us to manipulate the DOM are as follows: clear(): This will do nothing unless WebElement represents an input control. In this case, it will clear its value and then trigger a change event. It returns a promise that will be fulfilled whenever the operation is done. sendKeys(text or key, …): This will do nothing unless WebElement is an input control. In this case, it will send the equivalents of keyboard events to the parameters we have passed. It can receive one or more parameters with a text or key object. If it receives a text, it will transform the text into a sequence of keyboard events. This way, it will simulate a user typing on a keyboard. This is more realistic than simply changing the value property of an input control, since the proper keyDown, keyPress, and keyUp events will be fired. A promise is returned that will be fulfilled when all the key events are issued. For example, to simulate that a user enters some search text in an input field and then presses Enter, we can use the following code: var Key = require('selenium-webdriver').Key;   var searchField = driver.findElement({name: 'searchTxt'}); searchField.sendKeys('BDD with JS', Key.ENTER);  The webdriver.Key object allows us to specify any key that does not represent a character, such as Enter, the up arrow, Command, Ctrl, Shift, and so on. We can also use its chord method to represent a combination of several keys pressed at the same time. For example, to simulate Alt + Command + J, use driver.sendKeys(Key.chord(Key.ALT, Key.COMMAND, 'J'));. click(): This will issue a click event just in the center of the element. The returned promise will be fulfilled when the event is fired.  Sometimes, the center of an element is nonclickable, and an exception is thrown! This can happen, for example, with table rows, since the center of a table row may just be the padding between cells! submit(): This will look for the form that contains this element and will issue a submit event. Apart from sending events to an element, we can inspect its contents with the following methods: getId(): This will return a promise with the internal identifier of this element used by WebDriver. Note that this is not the value of the DOM ID property! getText(): This will return a promise that will be fulfilled with the visible text inside this element. It will include the text in any child element and will trim the leading and trailing whitespaces. Note that, if this element is not displayed or is hidden, the resulting text will be an empty string! getInnerHtml() and getOuterHtml(): These will return a promise that will be fulfilled with a string that contains innerHTML or outerHTML of this element. isSelected(): This will return a promise with a Boolean that determines whether the element has either been selected or checked. This method is designed to be used with the <option> elements. isEnabled(): This will return a promise with a Boolean that determines whether the element is enabled or not. isDisplayed(): This will return a promise with a Boolean that determines whether the element is displayed or not. Here, "displayed" is taken in a broad sense; in general, it means that the user can see the element without resizing the browser. For example, whether the element is hidden, whether it has diplay: none, or whether it has no size, or is in an inaccessible part of the document, the returned promise will be fulfilled as false. getTagName(): This will return a promise with the tag name of the element. getSize(): This will return a promise with the size of the element. The size comes as a JSON object with width and height properties that indicate the height and width in pixels of the bounding box of the element. The bounding box includes padding, margin, and border. getLocation(): This will return a promise with the position of the element. The position comes as a JSON object with x and y properties that indicate the coordinates in pixels of the element relative to the page. getAttribute(name): This will return a promise with the value of the specified attribute. Note that WebDriver does not distinguish between attributes and properties! If there is neither an attribute nor a property with that name, the promise will be fulfilled as null. If the attribute is a "boolean" HTML attribute (such as checked or disabled), the promise will be evaluated as true only if the attribute is present. If there is both an attribute and a property with the same name, the attribute value will be used.  If you really need to be precise about getting an attribute or a property, it is much better to use an injected script to get it. getCssValue(cssPropertyName): This will return a promise with a string that represents the computed value of the specified CSS property. The computed value is the resulting value after the browser has applied all the CSS rules and the style and class attributes. Note that the specific representation of the value depends on the browser; for example, the color property can be returned as red, #ff0000, or rgb(255, 0, 0) depending on the browser. This is not cross-browser, so we should avoid this method in our tests. findElement(locator) and findElements(locator): These will return an element, or all the elements that are the descendants of this element, and match the locator. isElementPresent(locator): This will return a promise with a Boolean that indicates whether there is at least one descendant element that matches this locator. As you can see, the WebElement API is pretty simple and allows us to do most of our tests easily. However, what if we need to perform some complex interaction with the UI, such as drag-and-drop? Complex UI interaction WebDriverJS allows us to define a complex action gesture in an easy way using the DSL defined in the webdriver.ActionSequence object. This DSL allows us to define any sequence of browser events using the builder pattern. For example, to simulate a drag-and-drop gesture, proceed with the following code: var beverageElement = driver.findElement({ id: 'expresso' });var orderElement = driver.findElement({ id: 'order' });driver.actions()    .mouseMove(beverageElement)    .mouseDown()    .mouseMove(orderElement)    .mouseUp()    .perform(); We want to drag an espresso to our order, so we move the mouse to the center of the espresso and press the mouse. Then, we move the mouse, by dragging the element, over the order. Finally, we release the mouse button to drop the espresso. We can add as many actions we want, but the sequence of events will not be executed until we call the perform method. The perform method will return a promise that will be fulfilled when the full sequence is finished. The webdriver.ActionSequence object has the following methods: sendKeys(keys...): This sends a sequence of key events, exactly as we saw earlier, to the method with the same name in the case of WebElement. The difference is that the keys will be sent to the document instead of a specific element. keyUp(key) and keyDown(key): These send the keyUp and keyDown events. Note that these methods only admit the modifier keys: Alt, Ctrl, Shift, command, and meta. mouseMove(targetLocation, optionalOffset): This will move the mouse from the current location to the target location. The location can be defined either as a WebElement or as page-relative coordinates in pixels, using a JSON object with x and y properties. If we provide the target location as a WebElement, the mouse will be moved to the center of the element. In this case, we can override this behavior by supplying an extra optional parameter indicating an offset relative to the top-left corner of the element. This could be needed in the case that the center of the element cannot receive events. mouseDown(), click(), doubleClick(), and mouseUp(): These will issue the corresponding mouse events. All of these methods can receive zero, one, or two parameters. Let's see what they mean with the following examples: var Button = require('selenium-webdriver').Button;   // to emit the event in the center of the expresso element driver.actions().mouseDown(expresso).perform(); // to make a right click in the current position driver.actions().click(Button.RIGHT).perform(); // Middle click in the expresso element driver.actions().click(expresso, Button.MIDDLE).perform();  The webdriver.Button object defines the three possible buttons of a mouse: LEFT, RIGHT, and MIDDLE. However, note that mouseDown() and mouseUp() only support the LEFT button! dragAndDrop(element, location): This is a shortcut to performing a drag-and-drop of the specified element to the specified location. Again, the location can be WebElement of a page-relative coordinate. Injecting scripts We can use WebDriver to execute scripts in the browser and then wait for its results. There are two methods for this: executeScript and executeAsyncScript. Both methods receive a script and an optional list of parameters and send the script and the parameters to the browser to be executed. They return a promise that will be fulfilled with the result of the script; it will be rejected if the script failed. An important detail is how the script and its parameters are sent to the browser. For this, they need to be serialized and sent through the network. Once there, they will be deserialized, and the script will be executed inside an autoexecuted function that will receive the parameters as arguments. As a result of of this, our scripts cannot access any variable in our tests, unless they are explicitly sent as parameters. The script is executed in the browser with the window object as its execution context (the value of this). When passing parameters, we need to take into consideration the kind of data that WebDriver can serialize. This data includes the following: Booleans, strings, and numbers. The null and undefined values. However, note that undefined will be translated as null. Any function will be transformed to a string that contains only its body. A WebElement object will be received as a DOM element. So, it will not have the methods of WebElement but the standard DOM method instead. Conversely, if the script results in a DOM element, it will be received as WebElement in the test. Arrays and objects will be converted to arrays and objects whose elements and properties have been converted using the preceding rules. With this in mind, we could, for example, retrieve the identifier of an element, such as the following one: var elementSelector = ".order ul > li"; driver.executeScript(     "return document.querySelector(arguments[0]).id;",     elementSelector ).then(function(id) {   expect(id).to.be.equal('order_item0'); }); Notice that the script is specified as a string with the code. This can be a bit awkward, so there is an alternative available: var elementSelector = ".order ul > li"; driver.executeScript(function() {     var selector = arguments[0];     return document.querySelector(selector).id; }, elementSelector).then(function(id) {   expect(id).to.be.equal('order_item0'); }); WebDriver will just convert the body of the function to a string and send it to the browser. Since the script is executed in the browser, we cannot access the elementSelector variable, and we need to access it through parameters. Unfortunately, we are forced to retrieve the parameters using the arguments pseudoarray, because WebDriver have no way of knowing the name of each argument. As its name suggest, executeAsyncScript allows us to execute an asynchronous script. In this case, the last argument provided to the script is always a callback that we need to call to signal that the script has finalized. The result of the script will be the first argument provided to that callback. If no argument or undefined is explicitly provided, then the result will be null. Note that this is not directly compatible with the Node.js callback convention and that any extra parameters passed to the callback will be ignored. There is no way to explicitly signal an error in an asynchronous way. For example, if we want to return the value of an asynchronous DAO, then proceed with the following code: driver.executeAsyncScript(function() {   var cb = arguments[1],       userId = arguments[0];   window.userDAO.findById(userId).then(cb, cb); }, 'user1').then(function(userOrError) {   expect(userOrError).to.be.equal(expectedUser); }); Command control flows All the commands in WebDriverJS are asynchronous and return a promise or WebElement. How do we execute an ordered sequence of commands? Well, using promises could be something like this: return driver.findElement({name:'quantity'}).sendKeys('23')     .then(function() {       return driver.findElement({name:'add'}).click();     })     .then(function() {       return driver.findElement({css:firstItemSel}).getText();     })     .then(function(quantity) {       expect(quantity).to.be.equal('23');     }); This works because we wait for each command to finish before issuing the next command. However, it is a bit verbose. Fortunately, with WebDriverJS we can do the following: driver.findElement({name:'quantity'}).sendKeys('23'); driver.findElement({name:'add'}).click(); return expect(driver.findElement({css:firstItemSel}).getText())     .to.eventually.be.equal('23'); How can the preceding code work? Because whenever we tell WebDriverJS to do something, it simply schedules the requested command in a queue-like structure called the control flow. The point is that each command will not be executed until it reaches the top of the queue. This way, we do not need to explicitly wait for the sendKeys command to be completed before executing the click command. The sendKeys command is scheduled in the control flow before click, so the latter one will not be executed until sendKeys is done. All the commands are scheduled against the same control flow queue that is associated with the WebDriver object. However, we can optionally create several control flows if we want to execute commands in parallel: var flow1 = webdriver.promise.createFlow(function() {   var driver = new webdriver.Builder().build();     // do something with driver here }); var flow2 = webdriver.promise.createFlow(function() {   var driver = new webdriver.Builder().build();     // do something with driver here }); webdriver.promise.fullyResolved([flow1, flow2]).then(function(){   // Wait for flow1 and flow2 to finish and do something }); We need to create each control flow instance manually and, inside each flow, create a separate WebDriver instance. The commands in both flows will be executed in parallel, and we can wait for both of them to be finalized to do something else using fullyResolved. In fact, we can even nest flows if needed to create a custom parallel command-execution graph. Taking screenshots Sometimes, it is useful to take some screenshots of the current screen for debugging purposes. This can be done with the takeScreenshot() method. This method will return a promise that will be fulfilled with a string that contains a base-64 encoded PNG. It is our responsibility to save this string as a PNG file. The following snippet of code will do the trick: driver.takeScreenshot()     .then(function(shot) {       fs.writeFileSync(fileFullPath, shot, 'base64');     });  Note that not all browsers support this capability. Read the documentation for the specific browser adapter to see if it is available. Working with several tabs and frames WebDriver allows us to control several tabs, or windows, for the same browser. This can be useful if we want to test several pages in parallel or if our test needs to assert or manipulate things in several frames at the same time. This can be done with the switchTo() method that will return a webdriver.WebDriver.TargetLocator object. This object allows us to change the target of our commands to a specific frame or window. It has the following three main methods: frame(nameOrIndex): This will switch to a frame with the specified name or index. It will return a promise that is fulfilled when the focus has been changed to the specified frame. If we specify the frame with a number, this will be interpreted as a zero-based index in the window.frames array. window(windowName): This will switch focus to the window named as specified. The returned promise will be fulfilled when it is done. alert(): This will switch the focus to the active alert window. We can dismiss an alert with driver.switchTo().alert().dismiss();. The promise returned by these methods will be rejected if the specified window, frame, or alert window is not found. To make tests on several tabs at the same time, we must ensure that they do not share any kind of state, or interfere with each other through cookies, local storage, or an other kind of mechanism. Summary This article showed us that a good way to test the UI of an application is actually to split it into two parts and test them separately. One part is the core logic of the UI that takes responsibility for control logic, models, calls to the server, validations, and so on. This part can be tested in a classic way, using BDD, and mocking the server access. No new techniques are needed for this, and the tests will be fast. Here, we can involve nonengineer stakeholders, such as UX designers, users, and so on, to write some nice BDD features using Gherkin and Cucumber.js. The other part is a thin view layer that follows a passive view design. It only updates the HTML when it is asked for, and listens to DOM events to transform them as requests to the core logic UI layer. This layer has no internal state or control rules; it simply transforms data and manipulates the DOM. We can use WebDriverJS to test the view. This is a good approach because the most complex part of the UI can be fully test-driven easily, and the hard and slow parts to test the view do not need many tests since they are very simple. In this sense, the passive view should not have a state; it should only act as a proxy of the DOM. Resources for Article: Further resources on this subject: Dart With Javascript [article] Behavior-Driven Development With Selenium WebDriver [article] Event-Driven Programming [article]

In this article, by Andrea Saccà, the author of Mastering Magento Theme Design, we will learn how to integrate the Bootstrap 3 framework and how to develop the main theme blocks. The following topics will be covered in this article: An introduction to Bootstrap Downloading Bootstrap (the current Version 3.1.1) Downloading and including jQuery Integrating the files into the theme Defining the main layout design template (For more resources related to this topic, see here.) An introduction to Bootstrap 3 Bootstrap is a sleek, intuitive, powerful, mobile-first frontend framework that enables faster and easier web development, as shown in the following screenshot: Bootstrap 3 is the most popular frontend framework that is used to create mobile-first websites. It includes a free collection of buttons, CSS components, and JavaScript to create websites or web applications; it was created by the Twitter team. Downloading Bootstrap (the current Version 3.1.1) First, you need to download the latest version of Bootstrap. The current version is 3.0. You can download the framework from http://getbootstrap.com/. The fastest way to download Bootstrap 3 is to download the precompiled and minified versions of CSS, JavaScript, and fonts. So, click on the Download Bootstrap button and unzip the file you downloaded. Once the archive is unzipped, you will see the following files: We need to take only the minified version of the files, that is, bootstrap.min.css from css, bootstrap.min.js from js, and all the files from font. For development, you can use bootstrap.css so that you can inspect the code and learn, and then switch to bootstrap.min.css when you go live. Copy all the selected files (CSS files inside the css folder, the .js files inside the js folder, and the font files inside the fonts folder) in the theme skin folder at skin/frontend/bookstore/default. Downloading and including jQuery Bootstrap is dependent on jQuery, so we have to download and include it before including boostrap.min.js. So, download jQuery from http://jquery.com/download/. The preceding URL takes us to the following screenshot: We will use the compressed production Version 1.10.2. Once you download jQuery, rename the file as jquery.min.js and copy it into the js skin folder at skin/frontend/bookstore/default/js/. In the same folder, also create the jquery.scripts.js file, where we will insert our custom scripts. Magento uses Prototype as the main JavaScript library. To make jQuery work correctly without conflicts, you need to insert the no conflict code in the jquery.scripts.js file, as shown in the following code: // This is important!jQuery.noConflict(); jQuery(document).ready(function() { // Insert your scripts here }); The following is a quick recap of CSS and JS files: Integrating the files into the theme Now that we have all the files, we will see how to integrate them into the theme. To declare the new JavaScript and CSS files, we have to insert the action in the local.xml file located at app/design/frontend/bookstore/default/layout. In particular, the file declaration needs to be done in the default handle to make it accessible by the whole theme. The default handle is defined by the following tags: <default> . . . </default> The action to insert the JavaScript and CSS files must be placed inside the reference head block. So, open the local.xml file and first create the following block that will define the reference: <reference name="head"> … </reference> Declaring the .js files in local.xml The action tag used to declare a new .js file located in the skin folder is as follows: <action method="addItem"> <type>skin_js</type><name>js/myjavascript.js</name> </action> In our skin folder, we copied the following three .js files: jquery.min.js jquery.scripts.js bootstrap.min.js Let's declare them as follows: <action method="addItem"> <type>skin_js</type><name>js/jquery.min.js</name> </action> <action method="addItem"> <type>skin_js</type><name>js/bootstrap.min.js</name> </action> <action method="addItem"> <type>skin_js</type><name>js/jquery.scripts.js</name> </action> Declaring the CSS files in local.xml The action tag used to declare a new CSS file located in the skin folder is as follows: <action method="addItem"> <type>skin_css</type><name>css/mycss.css</name> </action> In our skin folder, we have copied the following three .css files: bootstrap.min.css styles.css print.css So let's declare these files as follows: <action method="addItem"> <type>skin_css</type><name>css/bootstrap.min.css</name> </action> <action method="addItem"> <type>skin_css</type><name>css/styles.css</name> </action> <action method="addItem"> <type>skin_css</type><name>css/print.css</name> </action> Repeat this action for all the additional CSS files. All the JavaScript and CSS files that you insert into the local.xml file will go after the files declared in the base theme. Removing and adding the style.css file By default, the base theme includes a CSS file called styles.css, which is hierarchically placed before the bootstrap.min.css. One of the best practices to overwrite the Bootstrap CSS classes in Magento is to remove the default CSS files declared by the base theme of Magento, and declare it after Bootstrap's CSS files. Thus, the styles.css file loads after Bootstrap, and all the classes defined in it will overwrite the boostrap.min.css file. To do this, we need to remove the styles.css file by adding the following action tag in the xml part, just before all the css declaration we have already made: <action method="removeItem"> <type>skin_css</type> <name>css/styles.css</name> </action> Hence, we removed the styles.css file and added it again just after adding Bootstrap's CSS file (bootstrap.min.css): <action method="addItem"> <type>skin_css</type> <stylesheet>css/styles.css</stylesheet> </action> If it seems a little confusing, the following is a quick view of the CSS declaration: <!-- Removing the styles.css declared in the base theme --> <action method="removeItem"> <type>skin_css</type> <name>css/styles.css</name> </action> <!-- Adding Bootstrap Css --> <action method="addItem"> <type>skin_css</type> <stylesheet>css/bootstrap.min.css</stylesheet> </action> <!-- Adding the styles.css again --> <action method="addItem"> <type>skin_css</type> <stylesheet>css/styles.css</stylesheet> </action> Adding conditional JavaScript code If you check the Bootstrap documentation, you can see that in the HTML5 boilerplate template, the following conditional JavaScript code is added to make Internet Explorer (IE) HTML 5 compliant: <!--[if lt IE 9]> <script src = "https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"> </script> <script src = "https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"> </script> <![endif]--> To integrate them into the theme, we can declare them in the same way as the other script tags, but with conditional parameters. To do this, we need to perform the following steps: Download the files at https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js and https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js. Move the downloaded files into the js folder of the theme. Always integrate JavaScript through the .xml file, but with the conditional parameters as follows: <action method="addItem"> <type>skin_js</type><name>js/html5shiv.js</name> <params/><if>lt IE 9</if> </action> <action method="addItem"> <type>skin_js</type><name>js/respond.min.js</name> <params/><if>lt IE 9</if> </action> A quick recap of our local.xml file Now, after we insert all the JavaScript and CSS files in the .xml file, the final local.xml file should look as follows: <?xml version="1.0" encoding="UTF-8"?> <layout version="0.1.0"> <default translate="label" module="page"> <reference name="head"> <!-- Adding Javascripts --> <action method="addItem"> <type>skin_js</type> <name>js/jquery.min.js</name> </action> <action method="addItem"> <type>skin_js</type> <name>js/bootstrap.min.js</name> </action> <action method="addItem"> <type>skin_js</type> <name>js/jquery.scripts.js</name> </action> <action method="addItem"> <type>skin_js</type> <name>js/html5shiv.js</name> <params/><if>lt IE 9</if> </action> <action method="addItem"> <type>skin_js</type> <name>js/respond.min.js</name> <params/><if>lt IE 9</if> </action> <!-- Removing the styles.css --> <action method="removeItem"> <type>skin_css</type><name>css/styles.css</name> </action> <!-- Adding Bootstrap Css --> <action method="addItem"> <type>skin_css</type> <stylesheet>css/bootstrap.min.css</stylesheet> </action> <!-- Adding the styles.css --> <action method="addItem"> <type>skin_css</type> <stylesheet>css/styles.css</stylesheet> </action> </reference> </default> </layout> Defining the main layout design template A quick tip for our theme is to define the main template for the site in the default handle. To do this, we have to define the template into the most important reference, root. In a few words, the root reference is the block that defines the structure of a page. Let's suppose that we want to use a main structure having two columns with the left sidebar for the theme To change it, we should add the setTemplate action in the root reference as follows: <reference name="root"> <action method="setTemplate"> <template>page/2columns-left.phtml</template> </action> </reference> You have to insert the reference name "root" tag with the action inside the default handle, usually before every other reference. Defining the HTML5 boilerplate for main templates After integrating Bootstrap and jQuery, we have to create our HTML5 page structure for the entire base template. The following are the structure files that are located at app/design/frontend/bookstore/template/page/: 1column.phtml 2columns-left.phtml 2columns-right.phtml 3columns.phtml The Twitter Bootstrap uses scaffolding with containers, a row, and 12 columns. So, its page layout would be as follows: <div class="container"> <div class="row"> <div class="col-md-3"></div> <div class="col-md-9"></div> </div> </div> This structure is very important to create responsive sections of the store. Now we will need to edit the templates to change to HMTL5 and add the Bootstrap scaffolding. Let's look at the following 2columns-left.phtml main template file: <!DOCTYPE HTML> <html><head> <?php echo $this->getChildHtml('head') ?> </head> <body <?php echo $this->getBodyClass()?' class="'.$this->getBodyClass().'"':'' ?>> <?php echo $this->getChildHtml('after_body_start') ?> <?php echo $this->getChildHtml('global_notices') ?> <header> <?php echo $this->getChildHtml('header') ?> </header> <section id="after-header"> <div class="container"> <?php echo $this->getChildHtml('slider') ?> </div> </section> <section id="maincontent"> <div class="container"> <div class="row"> <?php echo $this->getChildHtml('breadcrumbs') ?> <aside class="col-left sidebar col-md-3"> <?php echo $this->getChildHtml('left') ?> </aside> <div class="col-main col-md-9"> <?php echo $this->getChildHtml('global_messages') ?> <?php echo $this->getChildHtml('content') ?> </div> </div> </div> </section> <footer id="footer"> <div class="container"> <?php echo $this->getChildHtml('footer') ?> </div> </footer> <?php echo $this->getChildHtml('before_body_end') ?> <?php echo $this->getAbsoluteFooter() ?> </body> </html> You will notice that I removed the Magento layout classes col-main, col-left, main, and so on, as these are being replaced by the Bootstrap classes. I also added a new section, after-header, because we will need it after we develop the home page slider. Don't forget to replicate this structure on the other template files 1column.phtml, 2columns-right.phtml, and 3columns.phtml, changing the columns as you need. Summary We've seen how to integrate Bootstrap and start the development of a Magento theme with the most famous framework in the world. Bootstrap is very neat, flexible, and modular, and you can use it as you prefer to create your custom theme. However, please keep in mind that it can be a big drawback on the loading time of the page. Following these techniques by adding the JavaScript and CSS classes via XML, you can allow Magento to minify them to speed up the loading time of the site. Resources for Article: Further resources on this subject: Integrating Twitter with Magento [article] Magento : Payment and shipping method [article] Magento: Exploring Themes [article]

IntroductionIn the first part of this series, we built a robust RSS news retrieval system using Weaviate, enabling us to fetch and store news articles efficiently. Now, in this second part, we're taking the next leap by exploring how to harness the power of Weaviate for similarity search and integrating it with LangChain. We will delve into the creation of a Streamlit application that performs real-time similarity search, contextual understanding, and dynamic context building. With the increasing demand for relevant and contextual information, this section will unveil the magic of seamlessly integrating various technologies to create an enhanced user experience.Before we dive into the exciting world of similarity search and context building, let's ensure you're equipped with the necessary tools. Familiarity with Weaviate, Streamlit, and Python will be essential as we explore these advanced concepts and create a dynamic application.Similarity Search and Weaviate IntegrationThe journey of enhancing news context retrieval doesn't end with fetching articles. Often, users seek not just relevant information, but also contextually similar content. This is where similarity search comes into play.Similarity search enables us to find articles that share semantic similarities with a given query. In the context of news retrieval, it's like finding articles that discuss similar events or topics. This functionality empowers users to discover a broader range of perspectives and relevant articles.Weaviate's core strength lies in its ability to perform fast and accurate similarity search. We utilize the perform_similarity_search function to query Weaviate for articles related to a given concept. This function returns a list of articles, each scored based on its relevance to the query.import weaviate from langchain.llms import OpenAI import datetime import pytz from dateutil.parser import parse davinci = OpenAI(model_name='text-davinci-003') def perform_similarity_search(concept):    """    Perform a similarity search on the given concept.    Args:    - concept (str): The term to search for, e.g., "Bitcoin" or "Ethereum"      Returns:    - dict: A dictionary containing the result of the similarity search    """    client = weaviate.Client("<http://weaviate:8080>")      nearText = {"concepts": [concept]}    response = (        client.query        .get("RSS_Entry", ["title", "link", "summary", "publishedDate", "body"])        .with_near_text(nearText)        .with_limit(50)  # fetching a maximum of 50 similar entries        .with_additional(['certainty'])        .do()    )      return response def sort_and_filter(results):    # Sort results by certainty    sorted_results = sorted(results, key=lambda x: x['_additional']['certainty'], reverse=True)    # Sort the top results by date    top_sorted_results = sorted(sorted_results[:50], key=lambda x: parse(x['publishedDate']), reverse=True)    # Return the top 10 results    return top_sorted_results[:5] # Define the prompt template template = """ You are a financial analysts reporting on latest developments and providing an overview about certain topics you are asked about. Using only the provided context, answer the following question. Prioritize relevance and clarity in your response. If relevant information regarding the query is not found in the context, clearly indicate this in the response asking the user to rephrase to make the search topics more clear. If information is found, summarize the key developments and cite the sources inline using numbers (e.g., [1]). All sources should consistently be cited with their "Source Name", "link to the article", and "Date and Time". List the full sources at the end in the same numerical order. Today is: {today_date} Context: {context} Question: {query} Answer: Example Answer (for no relevant information): "No relevant information regarding 'topic X' was found in the provided context." Example Answer (for relevant information): "The latest update on 'topic X' reveals that A and B have occurred. This was reported by 'Source Name' on 'Date and Time' [1]. Another significant development is D, as highlighted by 'Another Source Name' on 'Date and Time' [2]." Sources (if relevant): [1] Source Name, "link to the article provided in the context", Date and Time [2] Another Source Name, "link to the article provided in the context", Date and Time """ # Modified the generate_response function to now use the SQL agent def query_db(query):    # Query the weaviate database    results = perform_similarity_search(query)    results = results['data']['Get']['RSS_Entry']    top_results = sort_and_filter(results)    # Convert your context data into a readable string    context_string = [f"title:{r['title']}\\nsummary:{r['summary']}\\nbody:{r['body']}\\nlink:{r['link']}\\npublishedDate:{r['publishedDate']}\\n\\n" for r in top_results]    context_string = '\\n'.join(context_string)    # Get today's date    date_format = "%a, %d %b %Y %H:%M:%S %Z"    today_date = datetime.datetime.now(pytz.utc).strftime(date_format)    # Format the prompt    prompt = template.format(        query=query,        context=context_string,        today_date=today_date    )    # Print the formatted prompt for verification    print(prompt)    # Run the prompt through the model directly    response = davinci(prompt)    # Extract and print the response    return responseRetrieved results need effective organization for user consumption. The sort_and_filter function handles this task. It first sorts the results based on their certainty scores, ensuring the most relevant articles are prioritized. Then, it further sorts the top results by their published dates, providing users with the latest information to build the context for the LLM.LangChain Integration for Context BuildingWhile similarity search enhances content discovery, context is the key to understanding the significance of articles. Integrating LangChain with Weaviate allows us to dynamically build context and provide more informative responses.LangChain, a language manipulation tool, acts as our context builder. It enhances the user experience by constructing context around the retrieved articles, enabling users to understand the broader narrative. Our modified query_db function now incorporates Langchain's capabilities. The function generates a context-rich prompt that combines the user's query and the top retrieved articles. This prompt is structured using a template that ensures clarity and relevance.The prompt template is a structured piece of text that guides LangChain to generate contextually meaningful responses. It dynamically includes information about the query, context, and relevant articles. This ensures that users receive comprehensive and informative answers.Subsection 2.4: Handling Irrelevant Queries One of LangChain's unique strengths is its ability to gracefully handle queries with limited context. When no relevant information is found in the context, LangChain generates a response that informs the user about the absence of relevant data. This ensures transparency and guides users to refine their queries for better results.In the next section, we will be integrating this enhanced news retrieval system with a Streamlit application, providing users with an intuitive interface to access relevant and contextual information effortlessly.Building the Streamlit ApplicationIn the previous section, we explored the intricate layers of building a robust news context retrieval system using Weaviate and LangChain. Now, in this third part, we're diving into the realm of user experience enhancement by creating a Streamlit application. Streamlit empowers us to transform our backend functionalities into a user-friendly front-end interface with minimal effort. Let's discover how we can harness the power of Streamlit to provide users with a seamless and intuitive way to access relevant news articles and context.Streamlit is a Python library that enables developers to create interactive web applications with minimal code. Its simplicity, coupled with its ability to provide real-time visualizations, makes it a fantastic choice for creating data-driven applications.The structure of a Streamlit app is straightforward yet powerful. Streamlit apps are composed of simple Python scripts that leverage the provided Streamlit API functions. This section will provide an overview of how the Streamlit app is structured and how its components interact.import feedparser import pandas as pd import time from bs4 import BeautifulSoup import requests import random from datetime import datetime, timedelta import pytz import uuid import weaviate import json import time def wait_for_weaviate():    """Wait until Weaviate is available."""      while True:        try:            # Try fetching the Weaviate metadata without initiating the client here            response = requests.get("<http://weaviate:8080/v1/meta>")            response.raise_for_status()            meta = response.json()                      # If successful, the instance is up and running            if meta:                print("Weaviate is up and running!")                return        except (requests.exceptions.RequestException):            # If there's any error (connection, timeout, etc.), wait and try again            print("Waiting for Weaviate...")            time.sleep(5) RSS_URLS = [    "<https://thedefiant.io/api/feed>",    "<https://cointelegraph.com/rss>",    "<https://cryptopotato.com/feed/>",    "<https://cryptoslate.com/feed/>",    "<https://cryptonews.com/news/feed/>",    "<https://smartliquidity.info/feed/>",    "<https://bitcoinmagazine.com/feed>",    "<https://decrypt.co/feed>",    "<https://bitcoinist.com/feed/>",    "<https://cryptobriefing.com/feed>",    "<https://www.newsbtc.com/feed/>",    "<https://coinjournal.net/feed/>",    "<https://ambcrypto.com/feed/>",    "<https://www.the-blockchain.com/feed/>" ] def get_article_body(link):    try:        headers = {            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.3'}        response = requests.get(link, headers=headers, timeout=10)        response.raise_for_status()        soup = BeautifulSoup(response.content, 'html.parser')        paragraphs = soup.find_all('p')        # Directly return list of non-empty paragraphs        return [p.get_text().strip() for p in paragraphs if p.get_text().strip() != ""]    except Exception as e:        print(f"Error fetching article body for {link}. Reason: {e}")        return [] def parse_date(date_str):    # Current date format from the RSS    date_format = "%a, %d %b %Y %H:%M:%S %z"    try:        dt = datetime.strptime(date_str, date_format)        # Ensure the datetime is in UTC        return dt.astimezone(pytz.utc)    except ValueError:        # Attempt to handle other possible formats        date_format = "%a, %d %b %Y %H:%M:%S %Z"        dt = datetime.strptime(date_str, date_format)        return dt.replace(tzinfo=pytz.utc) def fetch_rss(from_datetime=None):    all_data = []    all_entries = []      # Step 1: Fetch all the entries from the RSS feeds and filter them by date.    for url in RSS_URLS:        print(f"Fetching {url}")        feed = feedparser.parse(url)        entries = feed.entries        print('feed.entries', len(entries))        for entry in feed.entries:            entry_date = parse_date(entry.published)                      # Filter the entries based on the provided date            if from_datetime and entry_date <= from_datetime:                continue            # Storing only necessary data to minimize memory usage            all_entries.append({                "Title": entry.title,                "Link": entry.link,                "Summary": entry.summary,                "PublishedDate": entry.published            })    # Step 2: Shuffle the filtered entries.    random.shuffle(all_entries)    # Step 3: Extract the body for each entry and break it down by paragraphs.    for entry in all_entries:        article_body = get_article_body(entry["Link"])        print("\\nTitle:", entry["Title"])        print("Link:", entry["Link"])        print("Summary:", entry["Summary"])        print("Published Date:", entry["PublishedDate"])        # Create separate records for each paragraph        for paragraph in article_body:            data = {                "UUID": str(uuid.uuid4()), # UUID for each paragraph                "Title": entry["Title"],                "Link": entry["Link"],                "Summary": entry["Summary"],                "PublishedDate": entry["PublishedDate"],                "Body": paragraph            }            all_data.append(data)    print("-" * 50)    df = pd.DataFrame(all_data)    return df def insert_data(df,batch_size=100):    # Initialize the batch process    with client.batch as batch:        batch.batch_size = 100        # Loop through and batch import the 'RSS_Entry' data        for i, row in df.iterrows():            if i%100==0:                print(f"Importing entry: {i+1}")  # Status update            properties = {                "UUID": row["UUID"],                "Title": row["Title"],                "Link": row["Link"],                "Summary": row["Summary"],                "PublishedDate": row["PublishedDate"],                "Body": row["Body"]            }            client.batch.add_data_object(properties, "RSS_Entry") if __name__ == "__main__":    # Wait until weaviate is available    wait_for_weaviate()    # Initialize the Weaviate client    client = weaviate.Client("<http://weaviate:8080>")    client.timeout_config = (3, 200)    # Reset the schema    client.schema.delete_all()    # Define the "RSS_Entry" class    class_obj = {        "class": "RSS_Entry",        "description": "An entry from an RSS feed",        "properties": [            {"dataType": ["text"], "description": "UUID of the entry", "name": "UUID"},            {"dataType": ["text"], "description": "Title of the entry", "name": "Title"},            {"dataType": ["text"], "description": "Link of the entry", "name": "Link"},            {"dataType": ["text"], "description": "Summary of the entry", "name": "Summary"},            {"dataType": ["text"], "description": "Published Date of the entry", "name": "PublishedDate"},            {"dataType": ["text"], "description": "Body of the entry", "name": "Body"}        ],        "vectorizer": "text2vec-transformers"    }    # Add the schema    client.schema.create_class(class_obj)    # Retrieve the schema    schema = client.schema.get()    # Display the schema    print(json.dumps(schema, indent=4))    print("-"*50)    # Current datetime    now = datetime.now(pytz.utc)    # Fetching articles from the last days    days_ago = 3    print(f"Getting historical data for the last {days_ago} days ago.")    last_week = now - timedelta(days=days_ago)    df_hist =  fetch_rss(last_week)    print("Head")    print(df_hist.head().to_string())    print("Tail")    print(df_hist.head().to_string())    print("-"*50)    print("Total records fetched:",len(df_hist))    print("-"*50)    print("Inserting data")    # insert historical data    insert_data(df_hist,batch_size=100)    print("-"*50)    print("Data Inserted")    # check if there is any relevant news in the last minute    while True:        # Current datetime        now = datetime.now(pytz.utc)        # Fetching articles from the last hour        one_min_ago = now - timedelta(minutes=1)        df =  fetch_rss(one_min_ago)        print("Head")        print(df.head().to_string())        print("Tail")        print(df.head().to_string())              print("Inserting data")        # insert minute data        insert_data(df,batch_size=100)        print("data inserted")        print("-"*50)        # Sleep for a minute        time.sleep(60)Streamlit apps rely on specific Python libraries and functions to operate smoothly. We'll explore the libraries used in our Streamlit app, such as streamlit, weaviate, and langchain, and discuss their roles in enabling real-time context retrieval.Demonstrating Real-time Context RetrievalAs we bring together the various elements of our news retrieval system, it's time to experience the magic firsthand by using the Streamlit app to perform real-time context retrieval.The Streamlit app's interface, showcasing how users can input queries and initiate similarity searches ensures a user-friendly experience, allowing users to effortlessly interact with the underlying Weaviate and LangChain-powered functionalities. The Streamlit app acts as a bridge, making complex interactions accessible to users through a clean and intuitive interface.The true power of our application shines when we demonstrate its ability to provide context for user queries and how LangChain dynamically builds context around retrieved articles and responses, creating a comprehensive narrative that enhances user understanding.ConclusionIn this second part of our series, we've embarked on the journey of creating an interactive and intuitive user interface using Streamlit. By weaving together the capabilities of Weaviate, LangChain, and Streamlit, we've established a powerful framework for context-based news retrieval. The Streamlit app showcases how the integration of these technologies can simplify complex processes, allowing users to effortlessly retrieve news articles and their contextual significance. As we wrap up our series, the next step is to dive into the provided code and experience the synergy of these technologies firsthand. Empower your applications with the ability to deliver context-rich and relevant information, bringing a new level of user experience to modern data-driven platforms.Through these two articles, we've embarked on a journey to build an intelligent news retrieval system that leverages cutting-edge technologies. We've explored the foundations of Weaviate, delved into similarity search, harnessed LangChain for context building, and created a Streamlit application to provide users with a seamless experience. In the modern landscape of information retrieval, context is key, and the integration of these technologies empowers us to provide users with not just data, but understanding. As you venture forward, remember that these concepts are stepping stones. Embrace the code, experiment, and extend these ideas to create applications that offer tailored and relevant experiences to your users.Author BioAlan Bernardo Palacio is a data scientist and an engineer with vast experience in different engineering fields. His focus has been the development and application of state-of-the-art data products and algorithms in several industries. He has worked for companies such as Ernst and Young, and Globant, and now holds a data engineer position at Ebiquity Media helping the company to create a scalable data pipeline. Alan graduated with a Mechanical Engineering degree from the National University of Tucuman in 2015, participated as the founder of startups, and later on earned a Master's degree from the faculty of Mathematics at the Autonomous University of Barcelona in 2017. Originally from Argentina, he now works and resides in the Netherlands.LinkedIn

(For more resources related to this topic, see here.) Much ado about snapshots (Intermediate) Snapshots are a fantastic feature of VMware Fusion because they allow you to roll the VM back in time to a previously saved state. Using snapshots is easy, but understanding how they work is important. Now, first things first. A snapshot is not a backup, but rather a way to either safely roll back in time or to keep multiple configurations of an OS but share the same basic configuration. The latter is very handy when building websites. For example, you can have one snapshot with IE7, another with IE8, another with IE9, another with IE10, and so on. A backup is a separate copy of the entire VM and/or its contents ("Your VM" and "Your Data") on a different disk or backup service. A snapshot is about rolling back in time on the same machine. If you took the snapshot when we finished installing Windows 7 but before upgrading to Windows 8, you can easily switch back and forth between Windows 7 and 8 by simply restoring the state. Let's see how. Getting ready Firstly, the VM doesn't have to be running, but it can be. The snapshot feature is powerful enough to work even when the VM is still running, but goes much faster if the virtual machine is powered off or suspended. We can use the snapshot we took when we finished installing Windows 7. If you didn't take a snapshot at that time, you can go ahead and take a new snapshot now by clicking on the Take button from the snapshot window. How to do it... Snapshots are best taken when a VM is powered off. It doesn't have to be, but your computer will complete the "Take Snapshot" operation much faster if the VM is powered off or suspended. Both fully powered off and suspended tasks are much faster because the VM isn't in motion when the snapshot is taken, allowing the operation to finish at a single stroke. Otherwise, the way the snapshot mechanism works when a VM is running is that once it finishes, it has to now gather the new data that changed from when the snapshot operation started. So, if it took five minutes to take a snapshot, it then has to gather itself up to date for those five minutes. That might take a minute. After that, it has to go back again to gather that last minute. If that minute takes 20 seconds, it has to then gather those 20 seconds again. This is made worse with the more things you're doing within the virtual machine. So, get it done in one motion by suspending or powering off the VM first. Launching the snapshot window and examining the tree The following sequence of steps is used to initiate the snapshot process: Click on the Snapshots button in the VM window and have a look at the snapshot interface. In my example, the following was the view of my "tree" right after we finished installing Windows 7: When I finished upgrading to Windows 8, I took another snapshot. This allows me to go back in time to both a fresh Windows 7 and Windows 8 installation, as shown in the following screenshot: Restoring a snapshot Having a TARDIS or DeLorean might be more fun, but for the rest of us, we can go back in time by restoring a snapshot. Let's go back to Windows 7 from our Windows 8 VM. Follow these steps: In the Snapshot Manager window, simply double-click on the base disk at the top of the tree to restore it. It will ask about saving the current state. Choose Save when prompted asks as shown in the following screenshot. You can rename the snapshot at any time from this window by right-clicking on the name and clicking on Get Info. After a few seconds, depending on the speed of your Mac, the older version should now show as the Current State, as shown in the following screenshot. If the VM was running, it should just show up now as being Windows 7. If you see a spinning wheel in the upper-right corner, that's the "disk cleanup" activity working in the background. You can use the VM while it's doing this; however, it might be a bit slow on disk access while it's cleaning up the disks. If the VM is suspended or powered off when restoring, the operation is much faster because the VM isn't changing/running. With this technique, you can switch between Windows 7 and Windows 8 with ease. How it works... In Fusion, all of the VM's files, are stored under Documents | Virtual Machines by default. Your C: drive in Windows is actually a series of files on the Mac named in sequence, with a .vmdk extension inside the Virtual Machines folder, as shown in the following screenshot. You can view the files by right-clicking on the VM and clicking on Show Package Contents from the Virtual Machines folder in the Finder. When you create a VM, it starts with one virtual disk (called the base disk). This virtual disk, or VMDK, is broken up into 2 GB "chunks" by default, but it can be one big chunk if desired. So, for a 20 GB disk, you end up with about 10 or 11 .vmdk files. This is for easy transport with drives that don't support large drives (such as MS-DOS/FAT32-formatted drives), and you may also have a performance benefit in certain cases. When you take a snapshot, the currently active VMDK goes into read-only mode, and a new VMDK is created. All writes go to the new VMDK, and reads happen from the original VMDK when the bits are there. Fusion is smart enough to keep track of what files are where; so, when the VM is running, Fusion is reading all of the snapshots in the current state's chain. A .vmdk file is thus named <disk_name>-<snapshot_number>-<slice>.vmdk. So, in my example, Virtual Disk is my disk name. (I could have customized and specified something different by performing a Custom Virtual Machine operation at the beginning.) I have three disks: Virtual Disk, Virtual Disk-000001, and Virtual Disk-000003. This means I have two snapshots and a base disk. (I took one snapshot and deleted it, which is why there's no Virtual Disk-000002). Each of those disks are of 60 GB capacity, so there are 31 slices. (s001 to s031). Each file starts around 300 KB and can grow to just over 2 GB. You can see where things can start to get confusing now. It gets even better when you have snapshots that are based on snapshots. You can have multiple snapshots with a common parent, which introduces a new concept in Fusion, that is, snapshot trees. Snapshots are also a great way to make sure something new isn't going to destroy your VM. So, if you are about to install some software that might be risky, take a snapshot. It's easy to roll back if something goes wrong. There's more... Snapshots are complicated, but there's great material out there by the gurus behind Fusion itself about how they work on a more technical level. More information You can read more about snapshots from Eric Tung, one of the original developers of Fusion (the blog is a bit old, but still completely accurate with respect to how snapshots work) at http://blogs.vmware.com/teamfusion/2008/11/vmware-fusion-3.html. A great article by Eric to dispel some of the confusion around snapshots and how to use them is available at http://blogs.vmware.com/teamfusion/2008/11/bonustip-snaps.html. One thing to note is that the more snapshots you have, the more effort the Mac has to make to "glue" them all together when you're running your virtual machine. As a rule of thumb, don't take snapshots and keep them around forever if you don't intend on rolling back to them regularly. Also, each snapshot can grow to be the size of the entire C: drive in Windows. Use them when necessary, but be aware of their performance and disk-usage costs. Summary In this article, we studied about snapshots and their usage. It shows you how to use this to make one virtual machine with both Windows 7 and Windows 8. Resources for Article: Further resources on this subject: An Introduction to VMware Horizon Mirage [Article] Windows 8 with VMware View [Article] Securing vCloud Using the vCloud Networking and Security App Firewall [Article]

(For more resources related to this topic, see here.) Adding more template files to your theme Let's say our site needed to display posts from a specific category differently from the rest of the site, or we needed the home page to work differently, or maybe we wanted to have more control over how search results or 404 pages were displayed. With template files, we can do all that. A search.php file for search results WordPress handles search results pretty well already. Let's see what's displayed in our theme if we try to search for example post (Note that we've now added a Search widgetto the right-hand side footer widget area to make this possible): As you can see, it' s using our index.php template file, so the heading reads This Month:.We'd rather make it more obvious that these are search results. Now let's see what happens if we search for something that can't be found: Again, the heading isn't great. Our theme gives the user a message telling them what's happened (which is coded into index.php as we'll see), but we could beef that up a bit,for example by adding a list of the most recent posts. Time for action – creating a search.php template file Let's create our search.php file and add some code to get it working in the way we'dlike it to: In your theme folder, make a copy of index.php and call it search.php. Find the following code near the top of the file: <h2 class="thisMonth embossed" style="color:#fff;">This Month:</h2> Edit the contents of the h2 element so the line of code now reads: <h2 class="thisMonth embossed" style="color:#fff;">Searchresults:</h2> Find the loop. This will begin with: <?php if (have_posts()) :?><?php while (have_posts()) : the_post();?> The first section of the loop displays any posts found by the search, leave this as itis. The second section of the loop specifies what happens if no search results are found. It's in the following lines of code: <?php else : ?><h2 class="center">Not Found</h2><p class="center">Sorry, but you are looking for somethingthat isn't here.</p><?php get_search_form(); ?><?php endif; ?> Underneath the line that reads <?php get_search_form(); ?> and before <?php endif; ?>, add the following lines of code: <?php endif; ?>, add the following lines of code:<h3>Latest articles:</h3><?php $query = new WP_Query( array ( 'post_type' => 'post', 'post_count' => '5' ) );while ( $query->have_posts() ) : $query->the_post(); ?><ul><li><a href="<?php the_permalink(); ?>"><?php the_title(); ?></a></li></ul><?php endwhile; ?> Save your search.php file and try searching for something which isn't included in the site. What just happened? We created a new template file called search.php, which will be used to display theresults of a site search. We then edited the heading to make it clearer, and added somecode to display the latest posts if the search had no results. We actually did something pretty advanced, we added a second loop inside our original loop. Let's have a look at the code we added after the search form: The function $query = new WP_Query() runs a new query on the database, based on the WordPress WP_Query function, which is the function you should use when running a loop inside the main loop. We gave WP_Query the following parameters: 'post_type' => 'post' – this ensures that the query will only look for posts, not for any other kind of content. 'post_count' => '5' – this tells WordPress how many posts to show. Five, in this case. We then output the title of each post with the php the_title() tag whichwe've already used higher up in the loop to display post titles. We wrapped this in a link inside a list item. The link uses the_permalink() to link to the blog post whose title is displayed. This is very similar to the main loop. Finally, we added endwhile() to stop this loop. This doesn't replace theendwhile() at the end of our main loop, which is higher up in the file. For more on WP_Query and how to use it to create multiple loops, see http://codex.wordpress.org/Class_Reference/ WP_Query. Let's have a look at what our users will see when they do a search now. First,a successful search: Next, an unsuccessful search: So that's how we set up a template file for search results. Our search page is only displayingtwo posts because that's all we have on our site. If there were more than five, it would justdisplay the five most recent. Now let's set one up to display some pages differently. Creating a custom page template In many themes, all pages will need the same basic layout and content, with the same sidebarsand footer and the same styling. But sometimes you may need some pages to look different. For example, you might want to use different sidebars in different pages, or you might wanta different layout. Here we'll look at the second of those two options. Time for action – creating a custom page template Imagine that you have some pages containing a lot of content which you want to displayacross the full width of the page, without the sidebar getting in the way. The way to handlethis is to create a page template which doesn't include the sidebar, and then select thatpage template when you're creating or editing those pages. Let's try it out. In the same folder as your other theme files, make a copy of your page.php file and call it page-no-sidebar.php. At the very top of the file, above the line reading <?php get_header(); ?>,insert the following code: <?php/*Template Name: Full width page without sidebar*/?> Find the following line of code: <div class="content left two-thirds"> Edit it so it reads: <div class="content left full"> Now find the line that reads <?php get_sidebar(); ?> and delete it. Save your file. What just happened? We created a new template file called page-no-sidebar.php, and edited it to displaycontent differently from our default page template: We edited the classes for our .content div, using the object-oriented approach to styling used by the layout-core.css file. This will apply styling for the .fullclass to this div, so that it displays a full width instead of two-thirds of its containing element. We removed the line calling the get_sidebar include, so that it won't be displayed on any pages using this template. The lines we added at the top are essential for WordPress to pick up on our page templateand make it available in the WordPress admin. Page editors will see a drop-down list of pagetemplates available, and the name we defined in the template is what they'll see in that list,as shown in the following screenshot: As you can see, in the Page Attributes box to the right-hand side, a new select box has appeared called Template. Our new page template is listed in that select box, along withDefault Template, which is page.php. Now we can try it out by assigning this template to a page and seeing how it looks.

MySQL Replication is a feature of the MySQL server that allows you to replicate data from one MySQL database server (called the master) to one or more MySQL database servers (slaves). MySQL Replication has been supported in MySQL for a very long time and is an extremely flexible and powerful technology. Depending on the configuration, you can replicate all databases, selected databases, or even selected tables within a database. In this article, by Alex Davies, author of High Availability MySQL Cookbook, we will cover: Designing a replication setup Configuring a replication master Configuring a replication slave without synchronizing data Configuring a replication slave and migrating data with a simple SQL dump Using LVM to reduce downtime on master when bringing a slave online Replication safety tricks Installing and Managing Multi Master Replication Manager(MMM) for MySQL High Availability is covered seperately. Replication is asynchronous, that is, the process of replication is not immediate and there is no guarantee that slaves have the same contents as the master (this is in contrast to MySQL Cluster). Designing a replication setup There are many ways to architect a MySQL Replication setup, with the number of options increasing enormously with the number of machines. In this recipe, we will look at the most common topologies and discuss the advantages and disadvantages of each, in order to show you how to select the appropriate design for each individual setup. Getting ready MySQL replication is simple. A server involved in a replication setup has one of following two roles: Master: Master MySQL servers write all transactions that change data to a binary log Slave: Slave MySQL servers connect to a master (on start) and download the transactions from the master's binary log, thereby applying them to the local server Slaves can themselves act as masters; the transactions that they apply from their master can be added in turn to their log as if they were made directly against the slave. Binary logs are binary files that contain details of every transaction that the MySQL server has executed. Running the server with the binary log enabled makes performance about 1 percent slower. The MySQL master creates binary logs in the forms name.000001, name.000002, and so on. Once a binary log reaches a defined size, it starts a new one. After a certain period of time, MySQL removes old logs. The exact steps for setting up both slaves and masters are covered in later recipes, but for the rest of this recipe it is important to understand that slaves contact masters to retrieve newer bits of the binary log, and to apply these changes to their local database. How to do it... There are several common architectures that MySQL replication can be used with. We will briefly mention and discuss benefits and problems with the most common designs, although we will explore in detail only designs that achieve high availability. Master and slave A single master with one or more slaves is the simplest possible setup. A master with one slave connected from the local network, and one slave connected via a VPN over the Internet, is shown in the following diagram: A setup such as this—with vastly different network connections from the different slaves to the master—will result in the two slaves having slightly different data. It is likely that the locally attached slave may be more up to date, because the latency involved in data transfers over the Internet (and any possible restriction on bandwidth) may slow down the replication process. This Master-Slave setup has the following common uses and advantages: A local slave for backups, ensuring that there is no massive increase in load during a backup period. A remote location—due to the asynchronous nature of MySQL replication, there is no great problem if the link between the master and the slave goes down (the slave will catch up when reconnected), and there is no significant performance hit at the master because of the slave. It is possible to run slightly different structures (such as different indexes) and focus a small number of extremely expensive queries at a dedicated slave in order to avoid slowing down the master. This is an extremely simple setup to configure and manage. A Master-Slave setup unfortunately has the following disadvantages: No automatic redundancy. It is common in setups such as this to use lower specification hardware for the slaves, which means that it may be impossible to "promote" a slave to a master in the case of an master failure. Write queries cannot be committed on the slave node. This means write transactions will have to be sent over the VPN to the master (with associated latency, bandwidth, and availability problems). Replication is equivalent to a RAID 1 setup, which is not an enormously efficient use of disk space (In the previous example diagram, each piece of data is written three times). Each slave does put a slight load on the master as it downloads its binary log. The number of slaves thus can't increase infinitely. Multi-master (active / active) Multi-master replication involves two MySQL servers, both configured as replication masters and slaves. This means that a transaction executed on one is picked up by the other, and vice versa, as shown in the following diagram: A SQL client connecting to the master on the left will execute a query, which will end up in that master's binary log. The master on the right will pick this query up and execute it. The same process, in reverse, occurs when a query is executed on the master on the right. While this looks like a fantastic solution, there are problems with this design: It is very easy for the data on the servers to become inconsistent due to the non-deterministic nature of some queries and "race conditions" where conflicting queries are executed at the same time on each node Recent versions of MySQL include various tricks to minimize the likelihood of these problems, but they are still almost inevitable in most real-world setups. It is extremely difficult to discover if this inconsistency exists, until it gets so bad that the replication breaks (because a replicated query can't be executed on the other node). This design is only mentioned here for completeness; it is often strongly recommended not to use it. Either use the next design, or if more than one "active" node is required, use one of the other high-availability techniques that are available but not covered in this article.

(For more resources related to this topic, see here.) Getting ready The following prerequisites have to be met before we can introduce the first Windows Server 2012 Domain Controller into the existing Active Directory domain: In order to add a Windows Server 2012 Domain Controller, the Forest Functional Level (FFL) must be Windows Server 2003. ADPREP is part of the domain controller process and the schema will get upgraded during this process. So the account must have the Schema and Enterprise admins privileges to install the first Windows Server 2012 Domain Controller. If there is a firewall between the new server and the existing domain controllers, make sure all the RPC high ports are open between these servers. The domain controller installation and replication can be controlled by a static or a range of RPC ports by modifying the registry on the domain controllers. The new Windows 2012 server's primary DNS IP address must be the IP address of an existing domain controller. The new server must be able to access the existing Active Directory domain and controllers by NetBIOS and Fully Qualified Domain Name (FQDN). If the new domain controller will be in a new site or in a new subnet, make sure to update the Active Directory Sites and Services with this information. In Windows Server 2012, domain controllers can be remotely deployed by using the Server Manager. The following recipe provides the step-by-step instructions on how to deploy a domain controller in an existing Active Directory environment. How to do it... Install and configure a Windows Server 2012. Join the new Windows Server 2012 to the existing Active Directory domain. Open Server Manager. Navigate to the All Servers group in the left-hand side pane. From the Server Name box, right-click on the appropriate server and select the Add Roles and Features option. You can also select Add Roles and Features from the Manage menu in the command bar. If the correct server is not listed here, you can manually add it from the Manage tab on the top right-hand side and select Add Server. Click on Next on the Welcome window. In the Select Installation Type window, select Role based or Feature based installation. Click on Next. In the Select destination server window, select Select a server from the server pool option and the correct server from the Server Pool box. Click on Next. On the Select server roles window, select Active Directory Domain Services. You will see a pop-up window to confirm the installation of Group Policy Management Tool. It is not required to install the administrative tools on a domain controller. However, this tool is required for the Group Policy Object management and administration. Click on Next. Click on Next in the Select features window. Click on Next on the Active Directory Domain Services window. In the Confirm Installation Selections window, select the Restart the destination server automatically if required option. In the pop-up window click on Yes to confirm the restart option and click on Install. This will begin the installation process. You will see the progress on the installation window itself. This window can be closed without interrupting the installation process. You can get the status update from the notification section in the command bar as shown in the following screenshot: The Post-deployment Configuration option needs to be completed after the Active Directory Domain Services role installation. This process will promote the new server as a domain controller. From the notification window, select Promote this server to a domain controller hyperlink. From the Deployment Configuration window, you should be able to: Install a new forest Install a new child domain Add an additional domain controller for an existing domain Specify alternative credentials for the domain controller promotion, and so on Since our goal is to install an additional domain controller to an existing domain, select the Add a domain controller to an existing domain option. Click on Next. In the Domain Controller Options window, you will see the following options: Domain Name System (DNS) server Global Catalog (GC) Read only Domain controller (RODC) Site name: Type the Directory Service Restore Mode (DSRM) password Select Domain Name System (DNS) server and Global Catalog (GC) checkboxes and provide the Directory Services Restore Mode (DSRM) password. Click on Next. Click on Next on the DNS Options window. In the Additional Options window you will see the following options: Install from media Replicate from Accept the default options unless you have technical reasons to modify these. Click on Next. In the Paths window, you can specify the AD Database, Log, and SYSVOL locations. Select the appropriate locations and then click on Next. Review the Microsoft Infrastructure Planning and Design (IPD) guides for best practices recommendations. For performance improvements, it is recommended to place database, log, and so on in separate drives. Click on Next on the Preparation Options window. During this process the Active Directory Schema and Domain Preparation will happen in the background. You should be able to review the selected option on the next screen. You can export these settings and configurations to a PowerShell script by clicking on the View Script option in the bottom-right corner of the screen. This script can be used for future domain controller deployments. Click on Next to continue with the installation. The prerequisite checking process will happen in the background. You will see the result in the Prerequisites Check window. This is a new enhancement in Windows Server 2012. Review the result and click on Install. The progress of the domain controller promotion will display on the Installation window. The following warning message will be displayed on the destination server before it restarts the server: You can review the %systemroot%debugdcpromo.log and %SystemRoot%debugnetsetup.log log files to get more information about DCPROMO and domain join-related issues. Summary Thus we learned the details of how to do Active Directory migration and its prerequisites, schema upgrade procedure, verification of the schema version, and installation of the Windows Server 2012 Domain Controller in the existing Windows Server 2008 and Server 2008 R2 domain. Resources for Article : Further resources on this subject: Migrating from MS SQL Server 2008 to EnterpriseDB [Article] Moving a Database from SQL Server 2005 to SQL Server 2008 in Three Steps [Article] Authoring an EnterpriseDB report with Report Builder 2.0 [Article]

 In this article by Sebastian Raschka, the author of Python Machine Learning, we will take a look at the concept of multilayer artificial neural networks, which was inspired by hypotheses and models of how the human brain works to solve complex problem tasks. (For more resources related to this topic, see here.) Although artificial neural networks gained a lot of popularity in the recent years, early studies of neural networks goes back to the 1940s, when Warren McCulloch and Walter Pitt first described the concept of how neurons may work. However, the decades that followed saw the first implementation of the McCulloch-Pitt neuron model, Rosenblatt's perceptron in the 1950s. Many researchers and machine learning practitioners slowly began to lose interest in neural networks, since no one had a good solution for the training of a neural network with multiple layers. Eventually, interest in neural networks was rekindled in 1986 when D.E. Rumelhart, G.E. Hinton, and R.J. Williams were involved in the discovery and popularization of the backpropagation algorithm to train neural networks more efficiently (Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (1986). Learning representations by back-propagating errors. Nature 323 (6088): 533–536). During the last decade, many more major breakthroughs have been made, known as deep learning algorithms. These can be used to create so-called feature detectors from unlabeled data to pre-train deep neural networks—neural networks that are composed of many layers. Neural networks are a hot topic not only in academic research but also in big technology companies such as Facebook, Microsoft, and Google. They invest heavily in artificial neural networks and deep learning research. Today, complex neural networks powered by deep learning algorithms are considered state of the art when it comes to solving complex problems, such as image and voice recognition. Introducing the multilayer neural network architecture In this section, we will connect multiple single neurons to a multilayer feed-forward neural network; this type of network is also called multilayer perceptron (MLP). The following figure illustrates the concept of an MLP consisting of three layers: one input layer, one hidden layer, and one output layer. The units in the hidden layer are fully connected to the input layer, and the output layer is fully connected to the hidden layer, respectively. As shown in the preceding diagram, we denote the ith activation unit in the jth layer as , and the activation units  and  are the bias units, which we set equal to 1. The activation of the units in the input layer is just its input plus the bias unit: Each unit in layer j is connected to all units in layer j + 1 via a weight coefficient; for example, the connection between unit a in layer j and unit b in layer j + 1 would be written as  . Note that the superscript i in  stands for the ith sample, not the ith layer; in the following paragraphs, we will often omit the superscript i for clarity. Activating a neural network via forward propagation In this section, we will describe the process of forward propagation to calculate the output of an MLP model. To understand how it fits into the context of learning an MLP model, let's summarize the MLP learning procedure in three simple steps: Starting at the input layer, we forward propagate the patterns of the training data through the network to generate an output. Based on the network's output, we calculate the error we want to minimize using a cost function, which we will describe later. We then backpropagate the error, find its derivative with respect to each weight in the network, and update the model. Finally, after we have repeated steps 1-3 for many epochs and learned the weights of the MLP, we use forward propagation to calculate the network output, and apply a threshold function to obtain the predicted class labels in the one-hot representation, which we described in the previous section. Now, let's walk through the individual steps of forward propagation to generate an output from the patterns in the training data. Since each unit in the hidden unit is connected to all units in the input layers, we first calculate the activation  as follows: Here, is the net input and  is the activation function, which has to be differentiable so as to learn the weights that connect the neurons using a gradient-based approach. To be able to solve complex problems such as image classification, we need non-linear activation functions in our MLP model, for example, the sigmoid (logistic) activation function: The sigmoid function is an "S"-shaped curve that maps the net input z onto a logistic distribution in the range 0 to 1, which passes the origin at z = 0.5 as shown in the following graph: Intuitively, we can think of the neurons in the MLP as logistic regression units that return values in the continuous range between 0 and 1. For purposes of code efficiency and readability, we will now write the activation in a more compact form using the concepts of basic linear algebra, which will allow us to vectorize our code implantation via NumPy rather than writing multiple nested and expensive Python for-loops: Here,  is our [m +1] x 1 dimensional feature vector for a sample  plus bias unit, and  is [m + 1] x h dimensional weight matrix where h is the number of hidden units in our neural network. After matrix-vector multiplication, we obtain the [m + 1] x 1 dimensional net input vector  . Furthermore, we can generalize this computation to all n samples in the training set: is now an n x [m + 1] matrix, and the matrix-matrix multiplication will result in an h x n dimensional net input matrix  . Finally, we apply the activation function g to each value in the net input matrix to get the h x n activation matrix  for the next layer (here, the output layer): Similarly, we can rewrite the activation of the output layer in the vectorized form: Here, we multiply the t x n matrix  (t is the number of output class labels) by the h x n dimensional matrix  to obtain the t x n dimensional matrix  (the columns in this matrix represent the outputs for each sample). Lastly, we apply the sigmoid activation function to obtain the continuous-valued output of our network: Classifying handwritten digits In this section, we will train our first multilayer neural network to classify handwritten digits from the popular MNIST dataset (Mixed National Institute of Standards and Technology database), which has been constructed by Yann LeCun and others (Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, November 1998) and serves as a popular benchmark dataset for machine learning algorithms. Obtaining the MNIST dataset The MNIST dataset is publicly available at http://yann.lecun.com/exdb/mnist/ and consists of these four parts: Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB unzipped, 60,000 samples) Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, 60,000 labels) Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB, 10,000 samples) Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB unzipped, 10,000 labels) In this section, we will only be working with a subset of MNIST. Thus, we only need to download the training set images and training set labels. After downloading the files, I recommend that you unzip the files using the Unix/Linux GZip tool from the terminal for efficiency, for example, using the following command in your local MNIST download directory or, alternatively, your favorite unarchiver tool if you are working with a Microsoft Windows machine: gzip *ubyte.gz -d The images are stored in byte form, and using the following function, we will read them into NumPy arrays, which we will use to train our MLP: >>> import os >>> import struct >>> import numpy as np >>> def load_mnist(path): ... labels_path = os.path.join(path, 'train-labels-idx1-ubyte') ... images_path = os.path.join(path, 'train-images-idx3-ubyte') ... with open(labels_path, 'rb') as lbpath: ... magic, n = struct.unpack('>II', lbpath.read(8)) ... labels = np.fromfile(lbpath, dtype=np.uint8) ... with open(images_path, 'rb') as imgpath: ... magic, num, rows, cols = struct.unpack( ... ">IIII", imgpath.read(16)) ... images = np.fromfile(imgpath, ... dtype=np.uint8).reshape(len(labels), 784) ... return images, labels The load_mnist function returns an n x m dimensional NumPy array (images), where n is the number of samples (60,000), and m is the number of features. The images in the MNIST dataset consist of 28 x 28 pixels, and each pixel is represented by a grayscale intensity value. Here, we unroll the 28 x 28 pixels into 1D row vectors, which represent the rows in our images array (784 per row or image). The load_mnist function returns a second array, labels, which contains the 60,000 class labels (integers 0-9) of the handwritten digits. The way we read in the image might seem a little strange at first: magic, n = struct.unpack('>II', lbpath.read(8)) labels = np.fromfile(lbpath, dtype=np.int8) To understand how these two lines of code work, let's take a look at the dataset description from the MNIST website: [offset] [type] [value] [description] 0000 32 bit integer 0x00000801(2049) magic number (MSB first) 0004 32 bit integer 60000 number of items 0008 unsigned byte ?? label 0009 unsigned byte ?? label ........ xxxx unsigned byte ?? label Using the two lines of the preceding code, we first read in the "magic number," which is a description of the file protocol as well as the "number of items" (n) from the file buffer, before we read the following bytes into a NumPy array using the fromfile method. The fmt parameter value >II that we passed as an argument to struct.unpack can be composed of two parts: >: Big-endian (defines the order in which a sequence of bytes is stored) I: Unsigned integer After executing the following code, we should have a label vector of 60,000 instances, that is, a 60,000 × 784 dimensional image matrix: >>> X, y = load_mnist('mnist') >>> print('Rows: %d, columns: %d' % (X.shape[0], X.shape[1])) Rows: 60000, columns: 784 To get a idea of what those images in MNIST look like, let's define a function that reshapes a 784-pixel sample from our feature matrix into the original 28 × 28 image that we can plot via matplotlib's imshow function: >>> import matplotlib.pyplot as plt >>> def plot_digit(X, y, idx): ... img = X[idx].reshape(28,28) ... plt.imshow(img, cmap='Greys', interpolation='nearest') ... plt.title('true label: %d' % y[idx]) ... plt.show() Now let's use the plot_digit function to display an arbitrary digit (here, the fifth digit) from the dataset: >>> plot_digit(X, y, 4) Implementing a multilayer perceptron In this section, we will implement the code of an MLP with one input, one hidden, and one output layer to classify the images in the MNIST dataset. I tried to keep the code as simple as possible. However, it may seem a little complicated at first. If you are not running the code from the IPython notebook, I recommend that you copy it to a Python script file in your current working directory, for example, neuralnet.py, which you can then import into your current Python session via this: from neuralnet import NeuralNetMLP Now, let's initialize a new 784-50-10 MLP, a neural network with 784 input units (n_features), 50 hidden units (n_hidden), and 10 output units (n_output): >>> nn = NeuralNetMLP(n_output=10, ... n_features=X.shape[1], ... n_hidden=50, ... l2=0.1, ... l1=0.0, ... epochs=800, ... eta=0.001, ... alpha=0.001, ... decrease_const=0.00001, ... shuffle=True, ... minibatches=50, ... random_state=1) l2: The  parameter for L2 regularization. This is used to decrease the degree of overfitting; equivalently, l1 is the  for L1 regularization. epochs: The number of passes over the training set. eta: The learning rate . alpha: A parameter for momentum learning used to add a factor of the previous gradient to the weight update for faster learning: (where t is the current time step or epoch). decrease_const: The decrease constant d for an adaptive learning rate  that decreases over time for better convergence . shuffle: Shuffle the training set prior to every epoch to prevent the algorithm from getting stuck in circles. minibatches: Splitting of the training data into k mini-batches in each epoch. The gradient is computed for each mini-batch separately instead of the entire training data for faster learning. Next, we train the MLP using 10,000 samples from the already shuffled MNIST dataset. Note that we only use 10,000 samples to keep the time for training reasonable (up to 5 minutes on standard desktop computer hardware). However, you are encouraged to use more training data for model fitting to increase the predictive accuracy: >>> nn.fit(X[:10000], y[:10000], print_progress=True) Epoch: 800/800 Similar to our earlier Adaline implementation, we save the cost for each epoch in a cost_ list, which we can now visualize, making sure that the optimization algorithm has reached convergence. Here, we plot only every 50th step to account for the 50 mini-batches (50 minibatches × 800 epochs): >>> import matplotlib.pyplot as plt >>> plt.plot(range(len(nn.cost_)//50), nn.cost_[::50], color='red') >>> plt.ylim([0, 2000]) >>> plt.ylabel('Cost') >>> plt.xlabel('Epochs') >>> plt.show() As we can see, the optimization algorithm converged after approximately 700 epochs. Now let's evaluate the performance of the model by calculating the prediction accuracy: >>> y_pred = nn.predict(X[:10000]) >>> acc = np.sum(y[:10000] == y_pred, axis=0) / 10000 >>> print('Training accuracy: %.2f%%' % (acc * 100)) Training accuracy: 97.60% As you can see, the model gets most of the training data right. But how does it generalize to data that it hasn't seen before during training? Let's calculate the test accuracy on 5,000 images that were not included in the training set: >>> y_pred = nn.predict(X[10000:15000]) >>> acc = np.sum(y[10000:15000] == y_pred, axis=0) / 5000 >>> print('Test accuracy: %.2f%%' % (acc * 100)) Test accuracy: 92.40% Summary Based on the discrepancy between the training and test accuracy, we can conclude that the model slightly overfits the training data. To decrease the degree of overfitting, we can change the number of hidden units or the values of the regularization parameters, or fit the model on more training data. Resources for Article: Further resources on this subject: Asynchronous Programming with Python[article] The Essentials of Working with Python Collections[article] Python functions – Avoid repeating code [article]

(For more resources related to this topic, see here.) In order to realize these goals, in this article, we'll be doing the following: Displaying a countdown timer on the screen Configuring fonts Creating a game attribute to count lives Using graphics to display information Counting collected actors Keeping track of the levels Prior to continuing with the development of our game, let's take a little time out to review what we have achieved so far, and also to consider some of the features that our game will need before it can be published. A review of our progress The gameplay mechanics are now complete; we have a controllable character in the form of a monkey, and we have some platforms for the monkey to jump on and traverse the scene. We have also introduced some enemy actors, the croc and the snake, and we have Aztec statues falling from the sky to create obstacles for the monkey. Finally, we have the fruit, all of which must be collected by the monkey in order to successfully complete the level. With regards to the scoring elements of the game, we're currently keeping track of a countdown timer (displayed in the debug console), which causes the scene to completely restart when the monkey runs out of time. When the monkey collides with an enemy actor, the scene is not reloaded, but the monkey is sent back to its starting point in the scene, and the timer continues to countdown. Planning ahead – what else does our game need? With the gameplay mechanics working, we need to consider what our players will expect to happen when they have completed the task of collecting all the fruits. As mentioned in the introduction to this article, our plan is to create additional, more difficult levels for the player to complete! We also need to consider what will happen when the game is over; either when the player has succeeded in collecting all the fruits, or when the player has failed to collect the fruits in the allocated time. The solution that we'll be implementing in this game is to display a message to advise the player of their success or failure, and to provide options for the player to either return to the main menu, or if the task was completed successfully, continue to the next level within the game. We need to implement a structure so that the game can keep track of information, such as how many lives the player has left and which level of the game is currently being played. Let's put some information on the screen so that our players can keep track of the countdown timer. Displaying a countdown timer on the screen We created a new scene behavior called Score Management, which contains the Decrement Countdown event, shown as follows: Currently, as we can see in the previous screenshot, this event decrements the Countdown attribute by a value of 1, every second. We also have a debug print instruction that displays the current value of Countdown in the debug console to help us, as game developers, keep track of the countdown. However, players of the game cannot see the debug console, so we need to provide an alternative means of displaying the amount of time that the player has to complete the level. Let's see how we can display that information on the screen for players of our game. Time for action – displaying the countdown timer on the screen Ensure that the Score Management scene behavior is visible: click on the Dashboard tab, select Scene Behaviors, and double-click on the Score Management icon in the main panel. Click + Add Event | Basics | When Drawing. Double-click on the title of the new Drawing event, and rename it to Display Countdown. Click on the Drawing section button in the instruction block palette. Drag a draw text anything at (x: 0 y: 0) block into the orange when drawing event block in the main panel. Enter the number 10 into the x: textbox and also enter 10 into the y: textbox. Click on the drop-down arrow in the textbox after draw text and select Text | Basics. Then click on the text & text block. In the first textbox in green, … & … block, enter the text COUNTDOWN: (all uppercase, followed by a colon). In the second textbox, after the & symbol, click on the drop-down arrow and select Basics, then click on the anything as text block. Click on the drop-down arrow in the … as text block, and select Number | Attributes | Countdown. Ensure that the new Display Countdown event looks like the following screenshot: Test the game. What just happened? When the game is played, we can now see in the upper-left corner of the screen, a countdown timer that represents the value of the Countdown attribute as it is decremented each second. First, we created a new Drawing event, which we renamed to Display Countdown, and then we added a draw text anything at (x: 0 y: 0) block, which is used to display the specified text in the required location on the screen. We set both the x: and y: coordinates for displaying the drawn text to 10 pixels, that is, 10 pixels from the left-hand side of the screen, and 10 pixels from the top of the screen. The next task was to add some text blocks that enabled us to display an appropriate message along with the value of the Countdown attribute. The text & text block enables us to concatenate, or join together, two separate pieces of text. The Countdown attribute is a number, so we used the anything as text block to convert the value of the Countdown attribute to text to ensure that it will be displayed correctly when the game is being played. In practice, we could have just located the Countdown attribute block in the Attributes section of the palette, and then dragged it straight into the text & text block. However, it is best practice to correctly convert attributes to the appropriate type, as required by the instruction block. In our case, the number attribute is being converted to text because it is being used in the text concatenation instruction block. If we needed to use a text value in a calculation, we would convert it to a number using an anything as number block. Configuring fonts We can see, when testing the game, that the font we have used is not very interesting; it's a basic font that doesn't really suit the style of the game! Stencyl allows us to specify our own fonts, so our next step is to import a font to use in our game. Time for action – specifying a font for use in our game Before proceeding with the following steps, we need to locate the fonts-of-afrikaAfritubu.TTF file. Place the file in a location where it can easily be located, and continue with the following steps: In the Dashboard tab, click on Fonts. In the main panel, click on the box containing the words This game contains no Fonts. Click here to create one. In the Name textbox of the Create New… dialog box, type HUD Font and click on the Create button. In the left-hand panel, click on the Choose… button next to the Font selector. Locate the file Afritubu.TTF and double-click on it to open it. Note that the main panel shows a sample of the new font. In the left-hand panel, change the Size option to 25. Important: save the game! Return to the Display Countdown event in the Score Management scene behavior. In the instruction block palette, click on the Drawing section button and then the Styles category button. Drag the set current font to Font block above the draw text block in the when drawing event. Click on the Font option in the set current font to Font block, and select Choose Font from the pop-up menu. Double-click on the HUD Font icon in the Choose a Font… dialog box. Test the game. Observe the countdown timer at the upper-left corner of the game. What just happened? We can see that the countdown timer is now being displayed using the new font that we have imported into Stencyl, as shown in the following screenshot: The first step was to create a new blank font in the Stencyl dashboard and to give it a name (we chose HUD Font), and then we imported the font file from a folder on our hard disk. Once we had imported the font file, we could see a sample of the font in the main panel. We then increased the size of the font using the Size option in the left-hand panel. That's all we needed to do in order to import and configure a new font in Stencyl! However, before progressing, we saved the game to ensure that the newly imported font will be available for the next steps. With our new font ready to use, we needed to apply it to our countdown text in the Display Countdown behavior. So, we opened up the behavior and inserted the set current font to Font style block. The final step was to specify which font we wanted to use, by clicking on the Font option in the font style block, and choosing the new font, HUD Font, which we configured in the earlier steps. Heads-Up Display (HUD) is often used in games to describe either text or graphics that is overlaid on the main game graphics to provide the player with useful information. Using font files in Stencyl Stencyl can use any TrueType font that we have available on our hard disk (files with the extension TTF); many thousands of fonts are available to download from the Internet free of charge, so it's usually possible to find a font that suits the style of any game that we might be developing. Fonts are often subject to copyright, so be careful to read any licensing agreements that are provided with the font file, and only download font files from reliable sources. Have a go hero When we imported the font into Stencyl, we specified a new font size of 25, but it is a straightforward process to modify further aspects of the font style, such as the color and other effects. Click on the HUD Font tab to view the font settings (or reopen the Font Editor from the Dashboards tab) and experiment with the font size, color, and other effects to find an appropriate style for the game. Take this opportunity to learn more about the different effects that are available, referring to Stencyl's online help if required. Remember to test the game to ensure that any changes are effective and the text is not difficult to read! Creating a game attribute to count lives Currently, our game never ends. As soon as the countdown reaches zero, the scene is restarted, or when the monkey collides with an enemy actor, the monkey is repositioned at the starting point in the scene. There is no way for our players to lose the game! In some genres of game, the player will never be completely eliminated; effectively, the same game is played forever. But in a platform game such as ours, the player typically will have a limited number of chances or lives to complete the required task. In order to resolve our problem of having a never-ending game, we need to keep track of the number of lives available to our player. So let's start to implement that feature right now by creating a game attribute called Lives! Time for action – creating a Lives game attribute Click on the Settings icon on the Stencyl toolbar at the top of the screen. In the left-hand panel of the Game Settings dialog box, click on the Attributes option. Click on the green Create New button. In the Name textbox, type Lives. In the Category textbox, change the word Default to Scoring. In the Type section, ensure that the currently selected option is Number. Change Initial Value to 3. Click on OK to confirm the configuration. We'll leave the Game Settings dialog box open, so that we can take a closer look. What just happened? We have created a new game attribute called Lives. If we look at the rightmost panel of the Game Settings dialog box that we left open on the screen, we can see that we have created a new heading entitled SCORING, and underneath the heading, there is a label icon entitled Lives, as shown in the following screenshot: The Lives item is a new game attribute that can store a number. The category name of SCORING that we created is not used within the game. We can't access it with the instruction blocks; it is there purely as a memory aid for the game developer when working with game attributes. When many game attributes are used in a game, it can become difficult to remember exactly what they are for, so being able to place them under specific headings can be helpful. Using game attributes The attributes we have used so far, such as the Countdown attribute that we created in the Score Management behavior, lose their values as soon as a different scene is loaded, or when the current scene is reloaded. Some game developers may refer to these attributes as local called Lives, so let's attributes, because they belong to the behavior in which they were created. Losing its value is fine when the attribute is just being used within the current scene; for example, we don't need to keep track of the countdown timer outside of the Jungle scene, because the countdown is reset each time the scene is loaded. However, sometimes we need to keep track of values across several scenes within a game, and this is when game attributes become very useful. Game attributes work in a very similar manner to local attributes. They store values that can be accessed and modified, but the main difference is that game attributes keep their values even when a different scene is loaded. Currently, the issue of losing attribute values when a scene is reloaded is not important to us, because our game only has one scene. However, when our players succeed in collecting all the fruits, we want the next level to be started without resetting the number of lives. So we need the number of lives to be remembered when the next scene is loaded. We've created a game attribute called Lives, so let's put it to good use. Time for action – decrementing the number of lives If the Game Settings dialog box is still open, click on OK to close it. Open the Manage Player Collisions actor behavior. Click on the Collides with Enemies event in the left-hand panel. Click on the Attributes section button in the palette. Click on the Game Attributes category button. Locate the purple set Lives to 0 block under the Number Setters subcategory and drag it into the orange when event so that it appears above the red trigger event RestartLevel in behavior Health for Self block. Click on the drop-down arrow in the set Lives to … block and select 0 - 0 in the Math section. In the left textbox of the … - … block, click on the drop-down arrow and select Game Attributes | Lives. In the right-hand textbox, enter the digit 1. Locate the print anything block in the Flow section of the palette, under the Debug category, and drag it below the set Lives to Lives – 1 block. In the print … block, click on the drop-down arrow and select Text | Basics | text & text. In the first empty textbox, type Lives remaining: (including the colon). Click on the drop-down arrow in the second textbox and select Basics | anything as text. In the … as text block, click on the drop-down arrow and select Number | Game Attributes | Lives. Ensure that the Collides with Enemies event looks like the following screenshot: Test the game; make the monkey collide with an enemy actor, such as the croc, and watch the debug console! What just happened? We have modified the Collides with Enemies event in the Manage Player Collisions behavior so that it decrements the number of lives by one when the monkey collides with an enemy actor, and the new value of Lives is shown in the debug console. This was achieved by using the purple game attribute setter and getter blocks to set the value of the Lives game attribute to its current value minus one. For example, if the value of Lives is 3 when the event occurs, Lives will be set to 3 minus 1, which is 2! The print … block was then used to display a message in the console, advising how many lives the player has remaining. We used the text & text block to join the text Lives remaining: together with the current value of the Lives game attribute. The anything as text block converts the numeric value of Lives to text to ensure that it will display correctly. Currently, the value of the Lives attribute will continue to decrease below 0, and the monkey will always be repositioned at its starting point. So our next task is to make something happen when the value of the Lives game attribute reaches 0! No more click-by-click steps! From this point onwards, click-by-click steps to modify behaviors and to locate and place each instruction block will not be specified! Instead, an overview of the steps will be provided, and a screenshot of the completed event will be shown towards the end of each Time for action section. The search facility, at the top of the instruction block palette, can be used to locate the required instruction block; simply click on the search box and type any part of the text that appears in the required block, then press the Enter key on the keyboard to display all the matching blocks in the block palette. Time for action – detecting when Lives reaches zero Create a new scene called Game Over — Select the Dashboard tab, select Scenes, and then select Click here to create a new Scene. Leave all the settings at their default configuration and click on OK. Close the tab for the newly created scene. Open the Manage Player Collisions behavior and click on the Collides with Enemies event to display the event's instruction blocks. Insert a new if block under the existing print block. Modify the if block to if Lives < 0. Move the existing block, trigger event RestartLevel in behavior Health for Self into the if Lives > 0 block. Insert an otherwise block below the if Lives > 0 block. Insert a switch to Scene and Crossfade for 0 secs block inside the otherwise block. Click on the Scene option in the new block, then click on Choose Scene and select the Game Over scene. Change the secs textbox to 0 (zero). Ensure that our modified Collides with Enemies event now looks like the following screenshot: Test the game; make the monkey run into an enemy actor, such as the croc, three times. What just happened? We have modified the Collides with Enemies event so that the value of the Lives game attribute is tested after it has been decremented, and the game will switch to the Game Over scene if the number of lives remaining is less than zero. If the value of Lives is greater than zero, the RestartLevel event in the monkey's Health behavior is triggered. However, if the value of Lives is not greater than zero, the instruction in the otherwise block will be executed, and this switches to the (currently blank) Game Over scene that we have created. If we review all the instructions in the completed Collides with Enemies event, and write them in English, the statement will be: When the monkey collides with an enemy, reduce the value of Lives by one and print the new value to the debug console. Then, if the value of Lives is more than zero, trigger the RestartLevel event in the monkey's Health behavior, otherwise switch to the Game Over scene. Before continuing, we should note that the Game Over scene has been created as a temporary measure to ensure that as we are in the process of developing the game, it's immediately clear to us (the developer) that the monkey has run out of lives. Have a go hero Change the Countdown attribute value to 30 — open the Jungle scene, click on the Behaviors button, then select the Score Management behavior in the left panel to see the attributes for this behavior. The following tasks in this Have a go hero session are optional — failure to attempt them will not affect future tutorials, but it is a great opportunity to put some of our newly learned skills to practice! In the section, Time for action – displaying the countdown timer on the screen, we learned how to display the value of the countdown timer on the screen during gameplay. Using the skills that we have acquired in this article, try to complete the following tasks: Update the Score Management behavior to display the number of lives at the upper-right corner of the screen, by adding some new instruction blocks to the Display Counter event. Rename the Display Counter event to Display HUD. Remove the print Countdown block from the Decrement Countdown event also found in the Score Management behavior. Right-click on the instruction block and review the options available in the pop-up menu! Remove the print Lives remaining: & Lives as text instruction block from the Collides with Enemies event in the Manage Player Collisions behavior. Removing debug instructions Why did we remove the debug print … blocks in the previous Have a go hero session? Originally, we added the debug blocks to assist us in monitoring the values of the Countdown attribute and Lives game attribute during the development process. Now that we have updated the game to display the required information on the screen, the debug blocks are redundant! While it would not necessarily cause a problem to leave the debug blocks where they are, it is best practice to remove any instruction blocks that are no longer in use. Also, during development, excessive use of debug print blocks can have an impact on the performance of the game; so it's a good idea to remove them as soon as is practical. Using graphics to display information We are currently displaying two on-screen pieces of information for players of our game: the countdown timer and the number of lives available. However, providing too much textual information for players can be distracting for them, so we need to find an alternative method of displaying some of the information that the player needs during gameplay. Rather than using text to advise the player how much time they have remaining to complete the level, we're going to display a timer bar on the screen. Time for action – displaying a timer bar Open the Score Management scene behavior and click on the Display HUD event. In the when drawing event, right-click on the blue block that draws the text for the countdown timer and select Activate / Deactivate from the pop-up menu. Note that the block becomes faded. Locate the draw rect at (x: 0 y: 0) with (w: 0 h: 0) instruction block in the palette, and insert it at the bottom of the when drawing event. Click on the draw option in the newly inserted block and change it to fill. Set both the x: and y: textboxes to 10. Set the width (w:) to Countdown x 10. Set the height (h:) to 10. Ensure that the draw text … block and the fill rect at … block in the Display HUD event appear as shown in the following screenshot (the draw text LIVES: … block may look different if the earlier Have a go hero section was attempted): Test the game! What just happened? We have created a timer bar that displays the amount of time remaining for the player to collect the fruit, and the timer bar reduces in size with the countdown! First, in the Display HUD event we deactivated, or disabled, the block that was drawing the textual countdown message, because we no longer want the text message to be displayed on the screen. The next step was to insert a draw rect … block that was configured to create a filled rectangle at the upper-left corner of the screen and with a width equal to the value of the Countdown timer multiplied by 10. If we had not multiplied the value of the countdown by 10, the timer bar would be very small and difficult to see (try it)! We'll be making some improvements to the timer bar later in this article. Activating and deactivating instruction blocks When we deactivate an instruction block, as we did in the Display HUD event, it no longer functions; it's completely ignored! However, the block remains in place, but is shown slightly faded, and if required, it can easily be reenabled by right-clicking on it and selecting the Activate / Deactivate option. Being able to activate and deactivate instruction blocks without deleting them is a useful feature — it enables us to try out new instructions, such as our timer bar, without having to completely remove blocks that we might want to use in the future. If, for example, we decided that we didn't want to use the timer bar, we could deactivate it and reactivate the draw text … block! Deactivated instruction blocks have no impact on the performance of a game; they are completely ignored during the game compilation process. Have a go hero The tasks in this Have a go hero session are optional; failure to attempt them will not affect future tutorials. Referring to Stencyl's online help if required at www.stencyl.com/help/, try to make the following improvements to the timer bar: Specify a more visually appealing color for the rectangle Make it thicker (height) so that it is easier to see when playing the game Consider drawing a black border (known as a stroke) around the rectangle Try to make the timer bar reduce in size smoothly, rather than in big steps Ask an independent tester for feedback about the changes and then modify the bar based on the feedback. To view suggested modifications together with comments, review the Display HUD event in the downloadable files that accompany this article. Counting collected actors With the number of lives being monitored and displayed for the player, and the timer bar in place, we now need to create some instructions that will enable our game to keep track of how many of the fruit actors have been collected, and to carry out the appropriate action when there is no fruit left to collect. Time for action – counting the fruit Open the Score Management scene behavior and create a new number attribute (not a game attribute) with the configuration shown in the following screenshot (in the block palette, click on Attributes, and then click on Create an Attribute…). Add a new when created event and rename it to Initialize Fruit Required. Add the required instruction blocks to the new when created event, so the Initialize Fruit Required event appears as shown in the following screenshot, carefully checking the numbers and text in each of the blocks' textboxes: Note that the red of group block in the set actor value … block cannot be found in the palette; it has been dragged into place from the orange for each … of group Collectibles block. Test the game and look for the Fruit required message in the debug console. What just happened? Before we can create the instructions to determine if all the fruit have been collected, we need to know how many fruit actors there are to collect. So we have created a new event that stores that information for us in a number attribute called Fruit Required and displays it in the debug console. We have created a for each … of group Collectibles block. This very useful looping block will repeat the instructions placed inside it for each member of the specified group that can be found in the current scene. We have specified the Collectibles group, and the instruction that we have placed inside the new loop is increment Fruit Required by 1. When the loop has completed, the value of the Fruit Required attribute is displayed in the debug console using a print … block. When constructing new events, it's good practice to insert print … blocks so we can be confident that the instructions achieve the results that we are expecting. When we are happy that the results are as expected, perhaps after carrying out further testing, we can remove the debug printing from our event. We have also introduced a new type of block that can set a value for an actor; in this case, we have set actor value Collected for … of group to false. This block ensures that each of the fruit actors has a value of Collected that is set to false each time the scene is loaded; remember that this instruction is inside the for each … loopso it is being carried out for every Collectible group member in the current scene. Where did the actor's Collected value come from? Well, we just invented it! The set actor value … block allows us to create an arbitrary value for an actor at any time. We can also retrieve that value at any time with a corresponding get actor value … block, and we'll be doing just that when we check to see if a fruit actor has been collected in the next section, Time for action – detecting when all the fruit are collected. Translating our instructions into English, results in the following statement: For each actor in the Collectibles group, that can be found in this scene, add the value 1 to the Fruit Required attribute and also set the actor's Collected value to false. Finally, print the result in the debug console. Note that the print … block has been placed after the for each … loop, so the message will not be printed for each fruit actor; it will appear just once, after the loop has completed! If we wish to prove to ourselves that the loop is counting correctly, we can edit the Jungle scene and add as many fruit actors as we wish. When we test the game, we can see that the number of fruit actors in the scene is correctly displayed in the debug console. We have designed a flexible set of instructions that can be used in any scene with any number of fruit actors, and which does not require us (as the game designer) to manually configure the number of fruit actors to be collected in that scene! Once again, we have made life easier for our future selves! Now that we have the attribute containing the number of fruit to be collected at the start of the scene, we can create the instructions that will respond when the player has successfully collected them all. Time for action – detecting when all fruits have been collected Create a new scene called Level Completed, with a Background Color of yellow. Leave all the other settings at their default configuration. Close the tab for the newly created scene. Return to the Score Management scene behavior, and create a new custom event by clicking on + Add Event | Advanced | Custom Event. In the left-hand panel, rename the custom event to Fruit Collected. Add the required instruction blocks to the new Fruit Collected event, so it appears as shown in the following screenshot, again carefully checking the parameters in each of the text boxes: Note that there is no space in the when FruitCollected happens custom event name. Save the game and open the Manage Player Collisions actor behavior. Modify the Collides with Collectibles event so it appears as shown in the following screenshot. The changes are listed in the subsequent steps: A new if get actor value Collected for … of group = false block has been inserted. The existing blocks have been moved into the new if … block. A set actor value Collected for … of group to true block has been inserted above the grow … block. A trigger event FruitCollected in behavior Score Management for this scene block has been inserted above the do after 0.5 seconds block. An if … of group is alive block has been inserted into the do after 0.5 seconds block, and the existing kill … of group block has been moved inside the newly added if … block. Test the game; collect several pieces of fruit, but not all of them! Examine the contents of the debug console; it may be necessary to scroll the console horizontally to read the messages. Continue to test the game, but this time collect all the fruit actors. What just happened? We have created a new Fruit Collected event in the Score Management scene behavior, which switches to a new scene when all the fruit actors have been collected, and we have also modified the Collides with Collectibles event in the Manage Player Collisions actor behavior in order to count how many pieces of fruit remain to be collected. When testing the game we can see that, each time a piece of fruit is collected, the new value of the Fruit Required attribute is displayed in the debug console, and when all the fruit actors have been collected, the yellow Level Completed scene is displayed. The first step was to create a blank Level Completed scene, which will be switched to when all the fruit actors have been collected. As with the Game Over scene that we created earlier in this article, it is a temporary scene that enables us to easily determine when the task of collecting the fruit has been completed successfully for testing purposes. We then created a new custom event called Fruit Collected in the Score Management scene behavior. This custom event waits for the FruitCollected event trigger to occur, and when that trigger is received, the Fruit Required attribute is decremented by 1 and its new value is displayed in the debug console. A test is then carried out to determine if the value of the Fruit Required attribute is equal to zero, and if it is equal to zero, the bright yellow, temporary Level Completed scene will be displayed! Our final task was to modify the Collides with Collectibles event in the Manage Player Collisions actor behavior. We inserted an if… block to test the collectible actor's Collected value; remember that we initialized this value to false in the previous section, Time for action – counting the fruit. If the Collected value for the fruit actor is still false, then it hasn't been collected yet, and the instructions contained within the if … block will be carried out. Firstly, the fruit actor's Collected value is set to false, which ensures that this event cannot occur again for the same piece of fruit. Next, the FruitCollected custom event in the Score Management scene behavior is triggered. Following that, the do after 0.5 seconds block is executed, and the fruit actor will be killed. We have also added an if … of group is alive check that is carried out before the collectible actor is killed. Because we are killing the actor after a delay of 0.5 seconds, it's good practice to ensure that the actor still exists before we try to kill it! In some games, it may be possible for the actor to be killed by other means during that very short 0.5 second delay, and if we try to kill an actor that does not exist, a runtime error may occur, that is, an error that happens while the game is being played. This may result in a technical error message being displayed to the player, and the game cannot continue; this is extremely frustrating for players, and they are unlikely to try to play our game again! Preventing multiple collisions from being detected A very common problem experienced by game designers, who are new to Stencyl, occurs when a collision between two actors is repeatedly detected. When two actors collide, all collision events that have been created with the purpose of responding to that collision will be triggered repeatedly until the collision stops occurring, that is, when the two actors are no longer touching. If, for example, we need to update the value of an attribute when a collision occurs, the attribute might be updated dozens or even hundreds of times in only a few seconds! In our game, we want collisions between the monkey actor and any single fruit actor to cause only a single update to the Fruit Required attribute. This is why we created the actor value Collected for each fruit actor, and this value is initialized to be false, not collected, by the Initialize Fruit Required event in the Score Management scene behavior. When the Collides with Collectibles event in Manage Player Collisions actor behavior is triggered, a test is carried out to determine if the fruit actor has already been collected, and if it has been collected, no further instructions are carried out. If we did not have this test, then the FruitCollected custom event would be triggered numerous times, and therefore the Fruit Required attribute would be decremented numerous times, causing the value of the Fruit Required attribute to reach zero almost instantly; all because the monkey collided with a single fruit actor! Using a Boolean value of True or False to carry out a test in this manner is often referred to by developers as using a flag or Boolean flag. Note that, rather than utilizing an actor value to record whether or not a fruit actor has been collected, we could have created a new attribute and initialized and updated the attribute in the same way that we initialized and updated the actor value. However, this would have required more effort to configure, and there is no perceptible impact on performance when using actor values in this manner. Some Stencyl users never use actor values (preferring to always use attributes instead), however, this is purely a matter of preference and it is at the discretion of the game designer which method to use. In order to demonstrate what happens when the actor value Collected is not used to determine whether or not a fruit actor has been collected, we can simply deactivate the set actor value Collected for … of group to true instruction block in the Collides with Collectibles event. After deactivating the block, run the game with the debug console open, and allow the monkey to collide with a single fruit actor. The Fruit Required attribute will instantly be decremented multiple times, causing the level to be completed after colliding with only one fruit actor! Remember to reactivate the set actor value … block before continuing! Keeping track of the levels As discussed in the introduction to this article, we're going to be adding an additional level to our game, so we'll need a method for keeping track of a player's progress through the game's levels. Time for action – adding a game attribute to record the level Create a new number game attribute called Level, with the configuration shown in the following screenshot: Save the game. What just happened? We have created a new game attribute called Level, which will be used to record the current level of the game. A game attribute is being used here, because we need to access this value in other scenes within our game; local attributes have their values reset whenever a scene is loaded, whereas game attributes' values are retained regardless of whether or not a different scene has been loaded. Fixing the never-ending game! We've finished designing and creating the gameplay for the Monkey Run game, and the scoring behaviors are almost complete. However, there is an anomaly with the management of the Lives game attribute. The monkey correctly loses a life when it collides with an enemy actor, but currently, when the countdown expires, the monkey is simply repositioned at the start of the level, and the countdown starts again from the beginning! If we leave the game as it is, the player will have an unlimited number of attempts to complete the level — that's not much of a challenge! Have a go hero (Recommended) In the Countdown expired event, which is found in the Health actor behavior, modify the test for the countdown so that it checks for the countdown timer being exactly equal to zero, rather than the current test, which is for the Countdown attribute being less than 1. We only want the ShowAngel event to be triggered once when the countdown equals exactly zero! (Recommended) Update the game so that the Show Angel event manages the complete process of losing a life, that is, either when a collision occurs between the monkey and an enemy, or when the countdown timer expires. A single event should deduct a life and restart the level. (Optional) If we look carefully, we can see that the countdown timer bar starts to grow backwards when the player runs out of time! Update the Display HUD event in the Score Management scene behavior, so that the timer bar is only drawn when the countdown is greater than zero. There are many different ways to implement the above modifications, so take some time and plan the recommended modifications! Test the game thoroughly to ensure that the lives are reduced correctly and the level restarts as expected, when the monkey collides with the enemy, and when the countdown expires. It would certainly be a good idea to review the download file for this session, compare it with your own solutions, and review each event carefully, along with the accompanying comment blocks. There are some useful tips in the example file, so do take the time to have a look! Summary Although our game doesn't look vastly different from when we started this article, we have made some very important changes. First, we implemented a text display to show the countdown timer, so that players of our game can see how much time they have remaining to complete the level. We also imported and configured a font and used the new font to make the countdown display more visually appealing. We then implemented a system of tracking the number of lives that the player has left, and this was our first introduction to learning how game attributes can store information that can be carried across scenes. The most visible change that we implemented in this article was to introduce a timer bar that reduces in size as the countdown decreases. Although very few instruction blocks were required to create the timer bar, the results are very effective, and are less distracting for the player than having to repeatedly look to the top of the screen to read a text display. The main challenge for players of our game is to collect all the fruit actors in the allocated time, so we created an initialization event to count the number of fruit actors in the scene. Again, this event has been designed to be reusable, as it will always correctly count the fruit actors in any scene. We also implemented the instructions to test when there are no more fruit actors to be collected, so the player can be taken to the next level in the game when they have completed the challenge. A very important skill that we learned while implementing these instructions was to use actor values as Boolean flags to ensure that collisions are counted only once. Finally, we created a new game attribute to keep track of our players' progress through the different levels in the game Resources for Article : Further resources on this subject: Introduction to Game Development Using Unity 3D [Article] 2D game development with Monkey [Article] Getting Started with GameSalad [Article]

  MySQL for Python Integrate the flexibility of Python and the power of MySQL to boost the productivity of your Python applications Implement the outstanding features of Python's MySQL library to their full potential See how to make MySQL take the processing burden from your programs Learn how to employ Python with MySQL to power your websites and desktop applications Apply your knowledge of MySQL and Python to real-world problems instead of hypothetical scenarios A manual packed with step-by-step exercises to integrate your Python applications with the MySQL database server Read more about this book (For more resources on Phython see here.) Hello() To create a function, we necessarily have to go back to the CREATE statement. As in a Python function definition, MySQL expects us to declare the name of the function as well as any arguments it requires. Unlike Python, MySQL also wants the type of data that will be received by the function. The beginning of a basic MySQL function definition looks like this: CREATE FUNCTION hello(s CHAR(20)) MySQL then expects to know what kind of data to return. Again, we use the MySQL data type definitions for this. RETURNS CHAR(50) This just tells MySQL that the function will return a character string of 50 characters or less. If the function will always perform the same task, it is best for the sake of performance to include the keyword DETERMINISTIC next. If the behavior of the function varies, use the keyword NON-DETERMINISTIC. If no keyword is set for the characteristic of the function, MySQL defaults to NON-DETERMINISTIC. You can learn more about the characteristic keywords used in function definitions at: http://dev.mysql.com/doc/refman/5.5/en/create-procedure.html Finally comes the meat of the function definition. Here we can set variables and perform any calculations that we want. For our basic definition, we will simply return a concatenated string: RETURN CONCAT('Hello, ', s, '!'); The function obviously concatenates the word 'Hello' with whatever argument is passed to it and appends an exclamation point at the end. To call it we use SELECT as with the other functions: mysql> SELECT hello('world') as Greeting; Capitalise() A function to capitalize every initial letter in a string follows the same pattern. The main point of the function is to walk through the string, character by character, and use UPPER() on every character that does not follow a letter. DELIMITER Obviously, we need a way to pass the entire function to MySQL without having any of the lines evaluated until we call it. To do this, we use the keyword DELIMITER. DELIMITER allows users to tell MySQL to evaluate lines that end in the character(s) we set. So the process for complex function definitions becomes: Change the delimiter. Pass the function with the usual semicolons to indicate the end of the line. Change the delimiter back to a semicolon. Call the function. The DELIMITER keyword allows us to specify more than one character as the line delimiter. So in order to ensure we don't need to worry about our code inadvertently conflicting with a line delimiter, let's make the delimiter @@: DELIMITER @@ The function definition From here, we are free to define a function to our specification. The definition line will read as follows: CREATE FUNCTION `Capitalise`(instring VARCHAR(1000)) The function will return a character string of similar length and variability: RETURNS VARCHAR(1000) When MySQL functions extend beyond the simplest calculations, such as hello(), MySQL requires us to specify the beginning and ending of the function. We do that with the keywords BEGIN and END. So let's begin the function: BEGIN Next, we need to declare our variables and their types using the keyword DECLARE: DECLARE i INT DEFAULT 1;DECLARE achar, imark CHAR(1);DECLARE outstring VARCHAR(1000) DEFAULT LOWER(instring); The DEFAULT keyword allows us to specify what should happen if outstring should fail for some reason. Next, we define a WHILE loop: WHILE i <= CHAR_LENGTH(instring) DO The WHILE loop obviously begins with a conditional statement based on the character length of instring. The resulting action begins with the keyword DO. From here, we set a series of variables and express what should happen where a character follows one of the following: blank space & '' _ ? ; : ! , - / ( . The operational part of the function looks like this: SET achar = SUBSTRING(instring, i, 1); SET imark = CASE WHEN i = 1 THEN ' ' ELSE SUBSTRING(instring, i - 1, 1) END CASE; IF imark IN (' ', '&', '''', '_', '?', ';', ':', '!', ',', '-', '/', '(', '.') THEN SET outstring = INSERT(outstring, i, 1, UPPER(achar)); END IF; SET i = i + 1; Much of this code is self-explanatory. It is worth noting, however, that the apodosis of any conditional in MySQL must end with the keyword END. In the case of IF, we use END IF. In the second SET statement, the keyword CASE is an evaluative keyword that functions similar to the try...except structure in Python. If the WHEN condition is met, the empty THEN apodosis is executed. Otherwise, the ELSE exception applies and the SUBSTRING function is run. The CASE structure ends with END CASE. MySQL will equally recognize the use of END instead. The subsequent IF clause evaluates whether imark, defined as the character before achar, is one of the declared characters. If it is, then that character in instring is replaced with its uppercase equivalent in outstring. After the IF clause is finished, the loop is incremented by one. After the entire string is processed, we then end the WHILE loop with: END WHILE; After the function's operations are completed, we return the value of outstring and indicate the end of the function: RETURN outstring;END@@ Finally, we must not forget to return the delimiter to a semicolon: DELIMITER ; It is worth noting that, instead of defining a function in a MySQL session we can define it in a separate file and load it on the fly with the SOURCE command. If we save the function to a file called capfirst.sql in a directory temp, we can source it relatively: We can also use: SOURCE /home/skipper/temp/capfirst.sql;

This article by Nathan Kozyra, author of Mastering Concurrency in Go, tackles one of the Internet's most famous and esteemed challenges and attempt to solve it with core Go packages. (For more resources related to this topic, see here.) We've built a few usable applications; things we can start with and leapfrog into real systems for everyday use. By doing so, we've been able to demonstrate the basic and intermediate-level patterns involved in Go's concurrent syntax and methodology. However, it's about time we take on a real-world problem—one that has vexed developers (and their managers and VPs) for a great deal of the early history of the Web. In addressing and, hopefully, solving this problem, we'll be able to develop a high-performance web server that can handle a very large volume of live, active traffic. For many years, the solution to this problem was solely to throw hardware or intrusive caching systems at the problem; so, alternately, solving it with programming methodology should excite any programmer. We'll be using every technique and language construct we've learned so far, but we'll do so in a more structured and deliberate way than we have up to now. Everything we've explored so far will come into play, including the following points: Creating a visual representation of our concurrent application Utilizing goroutines to handle requests in a way that will scale Building robust channels to manage communication between goroutines and the loop that will manage them Profiling and benchmarking tools (JMeter, ab) to examine the way our event loop actually works Timeouts and concurrency controls—when necessary—to ensure data and request consistency Attacking the C10K problem The genesis of the C10K problem is rooted in serial, blocking programming, which makes it ideal to demonstrate the strength of concurrent programming, especially in Go. When he asked this in 1999, for many server admins and engineers, serving 10,000 concurrent visitors was something that would be solved with hardware. The notion that a single server on common hardware could handle this type of CPU and network bandwidth without falling over seemed foreign to most. The crux of his proposed solutions relied on producing non-blocking code. Of course, in 1999, concurrency patterns and libraries were not widespread. C++ had some polling and queuing options available via some third-party libraries and the earliest predecessor to multithreaded syntaxes, later available through Boost and then C++11. Over the coming years, solutions to the problem began pouring in across various flavors of languages, programming design, and general approaches. Any performance and scalability problem will ultimately be bound to the underlying hardware, so as always, your mileage may vary. Squeezing 10,000 concurrent connections on a 486 processor with 500 MB of RAM will certainly be more challenging than doing so on a barebones Linux server stacked with memory and multiple cores. It's also worth noting that a simple echo server would obviously be able to assume more cores than a functional web server that returns larger amounts of data and accepts greater complexity in requests, sessions, and so on, as we'll be dealing with here. Failing of servers at 10,000 concurrent connections When the Web was born and the Internet commercialized, the level of interactivity was pretty minimal. If you're a graybeard, you may recall the transition from NNTP/IRC and the like and how extraordinarily rudimentary the Web was. To address the basic proposition of [page request] → [HTTP response], the requirements on a web server in the early 1990s were pretty lenient. Ignoring all of the error responses, header reading, and settings, and other essential (but unrelated to the in → out mechanism) functions, the essence of the early servers was shockingly simple, at least compared to the modern web servers. The first web server was developed by the father of the Web, Tim Berners-Lee. Developed at CERN (such as WWW/HTTP itself), CERN httpd handled many of the things you would expect in a web server today—hunting through the code, you'll find a lot of notation that will remind you that the very core of the HTTP protocol is largely unchanged. Unlike most technologies, HTTP has had an extraordinarily long shelf life. Written in C in 1990, it was unable to utilize a lot of concurrency strategies available in languages such as Erlang. Frankly, doing so was probably unnecessary—the majority of web traffic was a matter of basic file retrieval and protocol. The meat and potatoes of a web server were not dealing with traffic, but rather dealing with the rules surrounding the protocol itself. You can still access the original CERN httpd site and download the source code for yourself from http://www.w3.org/Daemon/. I highly recommend that you do so as both a history lesson and a way to look at the way the earliest web server addressed some of the earliest problems. However, the Web in 1990 and the Web when the C10K question was first posed were two very different environments. By 1999, most sites had some level of secondary or tertiary latency provided by third-party software, CGI, databases, and so on, all of which further complicated the matter. The notion of serving 10,000 flat files concurrently is a challenge in itself, but try doing so by running them on top of a Perl script that accesses a MySQL database without any caching layer; this challenge is immediately exacerbated. By the mid 1990s, the Apache web server had taken hold and largely controlled the market (by 2009, it had become the first server software to serve more than 100 million websites). Apache's approach was rooted heavily in the earliest days of the Internet. At its launch, connections were initially handled first in, first out. Soon, each connection was assigned a thread from the thread pool. There are two problems with the Apache server. They are as follows: Blocking connections can lead to a domino effect, wherein one or more slowly resolved connections could avalanche into inaccessibility Apache had hard limits on the number of threads/workers you could utilize, irrespective of hardware constraints It's easy to see the opportunity here, at least in retrospect. A concurrent server that utilizes actors (Erlang), agents (Clojure), or goroutines (Go) seems to fit the bill perfectly. Concurrency does not solve the C10k problem in itself, but it absolutely provides a methodology to facilitate it. The most notable and visible example of an approach to the C10K problem today is Nginx, which was developed using concurrency patterns, widely available in C by 2002 to address—and ultimately solve—the C10k problem. Nginx, today, represents either the #2 or #3 web server in the world, depending on the source. Using concurrency to attack C10K There are two primary approaches to handle a large volume of concurrent requests. The first involves allocating threads per connection. This is what Apache (and a few others) do. On the one hand, allocating a thread to a connection makes a lot of sense—it's isolated, controllable via the application's and kernel's context switching, and can scale with increased hardware. One problem for Linux servers—on which the majority of the Web lives—is that each allocated thread reserves 8 MB of memory for its stack by default. This can (and should) be redefined, but this imposes a largely unattainable amount of memory required for a single server. Even if you set the default stack size to 1 MB, we're dealing with a minimum of 10 GB of memory just to handle the overhead. This is an extreme example that's unlikely to be a real issue for a couple of reasons: first, because you can dictate the maximum amount of resources available to each thread, and second, because you can just as easily load balance across a few servers and instances rather than add 10 GB to 80 GB of RAM. Even in a threaded server environment, we're fundamentally bound to the issue that can lead to performance decreases (to the point of a crash). First, let's look at a server with connections bound to threads (as shown in the following diagram), and visualize how this can lead to logjams and, eventually, crashes: This is obviously what we want to avoid. Any I/O, network, or external process that can impose some slowdown can bring about that avalanche effect we talked about, such that our available threads are taken (or backlogged) and incoming requests begin to stack up. We can spawn more threads in this model, but as mentioned earlier, there are potential risks there too, and even this will fail to mitigate the underlying problem. Taking another approach In an attempt to create our web server that can handle 10,000 concurrent connections, we'll obviously leverage our goroutine/channel mechanism to put an event loop in front of our content delivery to keep new channels recycled or created constantly. For this example, we'll assume we're building a corporate website and infrastructure for a rapidly expanding company. To do this, we'll need to be able to serve both static and dynamic content. The reason we want to introduce dynamic content is not just for the purposes of demonstration—we want to challenge ourselves to show 10,000 true concurrent connections even when a secondary process gets in the way. As always, we'll attempt to map our concurrency strategy directly to goroutines and channels. In a lot of other languages and applications, this is directly analogous to an event loop, and we'll approach it as such. Within our loop, we'll manage the available goroutines, expire or reuse completed ones, and spawn new ones where necessary. In this example visualization, we show how an event loop (and corresponding goroutines) can allow us to scale our connections without employing too many hard resources such as CPU threads or RAM: The most important step for us here is to manage that event loop. We'll want to create an open, infinite loop to manage the creation and expiration of our goroutines and respective channels. As part of this, we will also want to do some internal logging of what's happening, both for benchmarking and debugging our application. Building our C10K web server Our web server will be responsible for taking requests, routing them, and serving either flat files or dynamic files with templates parsed against a few different data sources. As mentioned earlier, if we exclusively serve flat files and remove much of the processing and network latency, we'd have a much easier time with handling 10,000 concurrent connections. Our goal is to approach as much of a real-world scenario as we can—very little of the Web operates on a single server in a static fashion. Most websites and applications utilize databases, CDNs (Content Delivery Networks), dynamic and uncached template parsing, and so on. We need to replicate them whenever possible. For the sake of simplicity, we'll separate our content by type and filter them through URL routing, as follows: /static/[request]: This will serve request.html directly /template/[request]: This will serve request.tpl after its been parsed through Go /dynamic/[request][number]: This will also serve request.tpl and parse it against a database source's record By doing this, we should get a better mixture of possible HTTP request types that could impede the ability to serve large numbers of users simultaneously, especially in a blocking web server environment. You can find Go's exceptional library to generate safe data-driven templating at http://golang.org/pkg/html/template/. By safe, we're largely referring to the ability to accept data and move it directly into templates without worrying about the sort of injection issues that are behind a large amount of malware and cross-site scripting. For the database source, we'll use MySQL here, but feel free to experiment with other databases if you're more comfortable with them. Like the html/template package, we're not going to put a lot of time into outlining MySQL and/or its variants. Benchmarking against a blocking web server It's only fair to add some starting benchmarks against a blocking web server first so that we can measure the effect of concurrent versus nonconcurrent architecture. For our starting benchmarks, we'll eschew any framework, and we'll go with our old stalwart, Apache. For the sake of completeness here, we'll be using an Intel i5 3GHz machine with 8 GB of RAM. While we'll benchmark our final product on Ubuntu, Windows, and OS X here, we'll focus on Ubuntu for our example. Our localhost domain will have three plain HTML files in /static, each trimmed to 80 KB. As we're not using a framework, we don't need to worry about raw dynamic requests, but only about static and dynamic requests in addition to data source requests. For all examples, we'll use a MySQL database (named master) with a table called articles that will contain 10,000 duplicate entries. Our structure is as follows: CREATE TABLE articles ( article_id INT NOT NULL AUTO_INCREMENT, article_title VARCHAR(128) NOT NULL, article_text VARCHAR(128) NOT NULL, PRIMARY KEY (article_id) ) With ID indexes ranging sequentially from 0-10,000, we'll be able to generate random number requests, but for now, we just want to see what kind of basic response we can get out of Apache serving static pages with this machine. For this test, we'll use Apache's ab tool and then gnuplot to sequentially map the request time as the number of concurrent requests and pages; we'll do this for our final product as well, but we'll also go through a few other benchmarking tools for it to get some better details.   Apache's AB comes with the Apache web server itself. You can read more about it at http://httpd.apache.org/docs/2.2/programs/ab.html. You can download it for Linux, Windows, OS X, and more from http://httpd.apache.org/download.cgi. The gnuplot utility is available for the same operating systems at http://www.gnuplot.info/. So, let's see how we did it. Have a look at the following graph: Ouch! Not even close. There are things we can do to tune the connections available (and respective threads/workers) within Apache, but this is not really our goal. Mostly, we want to know what happens with an out-of-the-box Apache server. In these benchmarks, we start to drop or refuse connections at around 800 concurrent connections. More troubling is that as these requests start stacking up, we see some that exceed 20 seconds or more. When this happens in a blocking server, each request behind it is queued; requests behind that are similarly queued and the entire thing starts to fall apart. Even if we cannot hit 10,000 concurrent connections, there's a lot of room for improvement. While a single server of any capacity is no longer the way we expect a web server environment to be designed, being able to squeeze as much performance as possible out of that server, ostensibly with our concurrent, event-driven approach, should be our goal. Handling requests The Gorilla toolkit certainly makes this easier, but we should also know how to intercept the functionality to impose our own custom handler. Here is a simple web router wherein we handle and direct requests using a custom http.Server struct, as shown in the following code: var routes []string type customRouter struct { } func (customRouter) ServeHTTP(rw http.ResponseWriter, r *http.Request) { fmt.Println(r.URL.Path); } func main() { var cr customRouter; server := &http.Server { Addr: ":9000", Handler:cr, ReadTimeout: 10 * time.Second, WriteTimeout: 10 * time.Second, MaxHeaderBytes: 1 << 20, } server.ListenAndServe() } Here, instead of using a built-in URL routing muxer and dispatcher, we're creating a custom server and custom handler type to accept URLs and route requests. This allows us to be a little more robust with our URL handling. In this case, we created a basic, empty struct called customRouter and passed it to our custom server creation call. We can add more elements to our customRouter type, but we really don't need to for this simple example. All we need to do is to be able to access the URLs and pass them along to a handler function. We'll have three: one for static content, one for dynamic content, and one for dynamic content from a database. Before we go so far though, we should probably see what our absolute barebones, HTTP server written in Go, does when presented with the same traffic that we sent Apache's way. By old school, we mean that the server will simply accept a request and pass along a static, flat file. You could do this using a custom router as we did earlier, taking requests, opening files, and then serving them, but Go provides a much simpler mode to handle this basic task in the http.FileServer method. So, to get some benchmarks for the most basic of Go servers against Apache, we'll utilize a simple FileServer and test it against a test.html page (which contains the same 80 KB file that we had with Apache). As our goal with this test is to improve our performance in serving flat and dynamic pages, the actual specs for the test suite are somewhat immaterial. We'd expect that while the metrics will not match from environment to environment, we should see a similar trajectory. That said, it's only fair we supply the environment used for these tests; in this case, we used a MacBook Air with a 1.4 GHz i5 processor and 4 GB of memory. First, we'll do this with our absolute best performance out of the box with Apache, which had 850 concurrent connections and 900 total requests. The results are certainly encouraging as compared to Apache. Neither of our test systems were tweaked much (Apache as installed and basic FileServer in Go), but Go's FileServer handles 1,000 concurrent connections without so much as a blip, with the slowest clocking in at 411 ms. Apache has made a great number of strides pertaining to concurrency and performance options in the last five years, but to get there does require a bit of tuning and testing. The intent of this experiment is not intended to denigrate Apache, which is well tested and established. Instead, it's to compare the out-of-the-box performance of the world's number 1 web server against what we can do with Go. To really get a baseline of what we can achieve in Go, let's see if Go's FileServer can hit 10,000 connections on a single, modest machine out of the box: ab -n 10500 -c 10000 -g test.csv http://localhost:8080/a.html We will get the following output: Success! Go's FileServer by itself will easily handle 10,000 concurrent connections, serving flat, static content. Of course, this is not the goal of this particular project—we'll be implementing real-world obstacles such as template parsing and database access, but this alone should show you the kind of starting point that Go provides for anyone who needs a responsive server that can handle a large quantity of basic web traffic. Routing requests So, let's take a step back and look again at routing our traffic through a traditional web server to include not only our static content, but also the dynamic content. We'll want to create three functions that will route traffic from our customRouter:serveStatic():: read function and serve a flat file serveRendered():, parse a template to display serveDynamic():, connect to MySQL, apply data to a struct, and parse a template. To take our requests and reroute, we'll change the ServeHTTP method for our customRouter struct to handle three regular expressions. For the sake of brevity and clarity, we'll only be returning data on our three possible requests. Anything else will be ignored. In a real-world scenario, we can take this approach to aggressively and proactively reject connections for requests we think are invalid. This would include spiders and nefarious bots and processes, which offer no real value as nonusers.