How-To Tutorials

The core idea of a decorator is to transform some original function into another form. A decorator creates a kind of composite function based on the decorator and the original function being decorated. In this tutorial, we'll understand how decorators can be used as higher-order functions in Python. This article is an extract from the 2nd edition of the bestseller, Functional Python Programming, authored by Steven Lott.  Working with Decorator function A decorator function can be used in one of the two following ways: As a prefix that creates a new function with the same name as the base function as follows: @decorator def original_function(): pass As an explicit operation that returns a new function, possibly with a new name: def original_function(): pass original_function = decorator(original_function) These are two different syntaxes for the same operation. The prefix notation has the advantages of being tidy and succinct. The prefix location is more visible to some readers. The suffix notation is explicit and slightly more flexible. While the prefix notation is common, there is one reason for using the suffix notation: we may not want the resulting function to replace the original function. We mayt want to execute the following command that allows us to use both the decorated and the undecorated functions: new_function = decorator(original_function) This will build a new function, named new_function(), from the original function. Python functions are first-class objects. When using the @decorator syntax, the original function is no longer available for use. A decorator is a function that accepts a function as an argument and returns a function as the result. This basic description is clearly a built-in feature of the language. The open question then is how do we update or adjust the internal code structure of a function? The answer is we don't. Rather than messing about with the inside of the code, it's much cleaner to define a new function that wraps the original function. It's easier to process the argument values or the result and leave the original function's core processing alone. We have two phases of higher-order functions involved in defining a decorator; they are as follows: At definition time, a decorator function applies a wrapper to a base function and returns the new, wrapped function. The decoration process can do some one-time-only evaluation as part of building the decorated function. Complex default values can be computed, for example. At evaluation time, the wrapping function can (and usually does) evaluate the base function. The wrapping function can pre-process the argument values or post-process the return value (or both). It's also possible that the wrapping function may avoid calling the base function. In the case of managing a cache, for example, the primary reason for wrapping is to avoid expensive calls to the base function. Simple Decorator function Here's an example of a simple decorator: from functools import wraps from typing import Callable, Optional, Any, TypeVar, cast FuncType = Callable[..., Any] F = TypeVar('F', bound=FuncType) def nullable(function: F) -> F: @wraps(function) def null_wrapper(arg: Optional[Any]) -> Optional[Any]: return None if arg is None else function(arg) return cast(F, null_wrapper) We almost always want to use the functools.wraps() function to assure that the decorated function retains the attributes of the original function. Copying the __name__, and __doc__ attributes, for example, assures that the resulting decorated function has the name and docstring of the original function. The resulting composite function, defined as the null_wrapper() function in the definition of the decorator, is also a type of higher-order function that combines the original function, the function() callable object, in an expression that preserves the None values. Within the resulting null_wrapper() function, the original function callable object is not an explicit argument; it is a free variable that will get its value from the context in which the null_wrapper() function is defined. The decorator function's return value is the newly minted function. It will be assigned to the original function's name. It's important that decorators only return functions and that they don't attempt to process data. Decorators use meta-programming: a code that creates a code. The resulting null_wrapper() function, however, will be used to process the real data. Note that the type hints use a feature of a TypeVar to assure that the result of applying the decorator will be a an object that's a type of Callable. The type variable F is bound to the original function's type; the decorator's type hint claims that the resulting function should have the same type as the argument function. A very general decorator will apply to a wide variety of functions, requiring a type variable binding. Creating composite function We can apply our @nullable decorator to create a composite function as follows: @nullable def nlog(x: Optional[float]) -> Optional[float]: return math.log(x) This will create a function, nlog(), which is a null-aware version of the built-in math.log() function. The decoration process returned a version of the null_wrapper() function that invokes the original nlog(). This result will be named nlog(), and will have the composite behavior of the wrapping and the original wrapped function. We can use this composite nlog() function as follows: >>> some_data = [10, 100, None, 50, 60] >>> scaled = map(nlog, some_data) >>> list(scaled) [2.302585092994046, 4.605170185988092, None, 3.912023005428146, 4.0943445622221] We've applied the function to a collection of data values. The None value politely leads to a None result. There was no exception processing involved. This type of example isn't really suitable for unit testing. We'll need to round the values for testing purposes. For this, we'll need a null-aware round() function too. Here's how we can create a null-aware rounding function using decorator notation: @nullable def nround4(x: Optional[float]) -> Optional[float]: return round(x, 4) This function is a partial application of the round() function, wrapped to be null-aware. In some respects, this is a relatively sophisticated bit of functional programming that's readily available to Python programmers. The typing module makes it particularly easy to describe the types of null-aware function and null-aware result, using the Optional type definition. The definition Optional[float] means Union[None, float]; either a None object or a float object may be used. We could also create the null-aware rounding function using the following code: nround4 = nullable(lambda x: round(x, 4)) Note that we didn't use the decorator in front of a function definition. Instead, we applied the decorator to a function defined as a lambda form.  This has the same effect as a decorator in front of a function definition. We can use this round4() function to create a better test case for our nlog() function as follows: >>> some_data = [10, 100, None, 50, 60] >>> scaled = map(nlog, some_data) >>> [nround4(v) for v in scaled] [2.3026, 4.6052, None, 3.912, 4.0943] This result will be independent of any platform considerations. It's very handy for doctest testing. It can be challenging to apply type hints to lambda forms. The following code shows what is required: nround4l: Callable[[Optional[float]], Optional[float]] = ( nullable(lambda x: round(x, 4)) ) The variable nround4l is given a type hint of Callable with an argument list of [Optional[float]] and a return type of Optional[float]. The use of the Callable hint is appropriate only for positional arguments. In cases where there will be keyword arguments or other complexities, see http://mypy.readthedocs.io/en/latest/kinds_of_types.html#extended-callable-types. The @nullable decorator makes an assumption that the decorated function is unary. We would need to revisit this design to create a more general-purpose null-aware decorator that works with arbitrary collections of arguments. If you found this tutorial useful and interested to learn more such techniques, grab the Steven Lott's bestseller, Functional Python Programming. Putting the Fun in Functional Python Why functional programming in Python matters: Interview with best selling author, Steven Lott Expert Python Programming: Interfaces

(For more resources related to this topic, see here.) OpenStack is a complex suite of software that can make tracking down issues and faults quite daunting to beginners and experienced system administrators alike. While there is no single approach to troubleshooting systems, understanding where OpenStack logs vital information or what tools are available to help track down bugs will help resolve issues we may encounter. However, OpenStack like all software will have bugs that we are not able to solve ourselves. In that case, we will show you how gathering the required information so that the OpenStack community can identify bugs and suggest fixes is important in ensuring those bugs or issues are dealt with quickly and efficiently. Understanding logging Logging is important in all computer systems, but the more complex the system, the more you rely on logging to be able to spot problems and cut down on troubleshooting time. Understanding logging in OpenStack is important to ensure your environment is healthy and you are able to submit relevant log entries back to the community to help fix bugs. Getting ready Log in as the root user onto the appropriate servers where the OpenStack services are installed. This makes troubleshooting easier as root privileges are required to view all the logs. How to do it... OpenStack produces a large number of logs that help troubleshoot our OpenStack installations. The following details outline where these services write their logs: OpenStack Compute services logs Logs for the OpenStack Compute services are written to /var/log/nova/, which is owned by the nova user, by default. To read these, log in as the root user (or use sudo privileges when accessing the files). The following is a list of services and their corresponding logs. Note that not all logs exist on all servers. For example, nova-compute.log exists on your compute hosts only: nova-compute: /var/log/nova/nova-compute.log Log entries regarding the spinning up and running of the instances nova-network: /var/log/nova/nova-network.log Log entries regarding network state, assignment, routing, and security groups nova-manage: /var/log/nova/nova-manage.log Log entries produced when running the nova-manage command nova-conductor: /var/log/nova/nova-conductor.log Log entries regarding services making requests for database information nova-scheduler: /var/log/nova/nova-scheduler.log Log entries pertaining to the scheduler, its assignment of tasks to nodes, and messages from the queue nova-api: /var/log/nova/nova-api.log Log entries regarding user interaction with OpenStack as well as messages regarding interaction with other components of OpenStack nova-cert: /var/log/nova/nova-cert.log Entries regarding the nova-cert process nova-console: /var/log/nova/nova-console.log Details about the nova-console VNC service nova-consoleauth: /var/log/nova/nova-consoleauth.log Authentication details related to the nova-console service nova-dhcpbridge: /var/log/nova/nova-dhcpbridge.log Network information regarding the dhcpbridge service OpenStack Dashboard logs OpenStack Dashboard (Horizon) is a web application that runs through Apache by default, so any errors and access details will be in the Apache logs. These can be found in /var/log/apache2/*.log, which will help you understand who is accessing the service as well as the report on any errors seen with the service. OpenStack Storage logs OpenStack Object Storage (Swift) writes logs to syslog by default. On an Ubuntu system, these can be viewed in /var/log/syslog. On other systems, these might be available at /var/log/messages. The OpenStack Block Storage service, Cinder, will produce logs in /var/log/cinder by default. The following list is a breakdown of the log files: cinder-api: /var/log/cinder/cinder-api.log Details about the cinder-api service cinder-scheduler: /var/log/cinder-scheduler.log Details related to the operation of the Cinder scheduling service cinder-volume: /var/log/cinder/cinder-volume.log Log entries related to the Cinder volume service OpenStack Identity logs The OpenStack Identity service, Keystone, writes its logging information to /var/log/keystone/keystone.log. Depending on how you have Keystone configured, the information in this log file can be very sparse to extremely verbose including complete plaintext requests. OpenStack Image Service logs The OpenStack Image Service Glance stores its logs in /var/log/glance/*.log with a separate log file for each service. The following is a list of the default log files: api: /var/log/glance/api.log Entries related to the glance API registry: /var/log/glance/registry.log Log entries related to the Glance registry service. Things like metadata updates and access will be stored here depending on your logging configuration. OpenStack Network Service logs OpenStack Networking Service, formerly Quantum, now Neutron, stores its log files in /var/log/quantum/*.log with a separate log file for each service. The following is a list of the corresponding logs: dhcp-agent: /var/log/quantum/dhcp-agent.log Log entries pertaining to the dhcp-agent l3-agent: /var/log/quantum/l3-agent.log Log entries related to the l3 agent and its functionality metadata-agent: /var/log/quantum/metadata-agent.log This file contains log entries related to requests Quantum has proxied to the Nova metadata service. openvswitch-agent: /var/log/quantum/openvswitch-agent.log Entries related the the operation of Open vSwitch. When implementing OpenStack Networking, if you use a different plugin, its log file will be named accordingly. server: /var/log/quantum/server.log Details and entries related to the quantum API service OpenVSwitch Server: /var/log/openvswitch/ovs-vswitchd.log Details and entries related to the OpenVSwitch Switch Daemon Changing log levels By default each OpenStack service has a sane level of logging, which is determined by the level set as Warning. That is, it will log enough information to provide you the status of the running system as well as some basic troubleshooting information. However, there will be times that you need to adjust the logging verbosity either up or down to help diagnose an issue or reduce logging noise. As each service can be configured similarly, we will show you how to make these changes on the OpenStack Compute service. Log-level settings in OpenStack Compute services To do this, log into the box where the OpenStack Compute service is running and execute the following commands: sudo vim /etc/nova/logging.conf Change the following log levels to either DEBUG, INFO or WARNING in any of the services listed: Log-level settings in other OpenStack services Other services such as Glance and Keystone currently have their log-level settings within their main configuration files such as /etc/glance/glance-api.conf. Adjust the log levels by altering the following lines to achieve INFO or DEBUG levels: Restart the relevant service to pick up the log-level change. How it works... Logging is an important activity in any software, and OpenStack is no different. It allows an administrator to track down problematic activity that can be used in conjunction with the community to help provide a solution. Understanding where the services log and managing those logs to allow someone to identify problems quickly and easily are important. Checking OpenStack services OpenStack provides tools to check on its services. In this section, we'll show you how to check the operational status of these services. We will also use common system commands to check whether our environment is running as expected. Getting ready To check our OpenStack Compute host, we must log into that server, so do this now before following the given steps. How to do it... To check that OpenStack Compute is running the required services, we invoke the nova-manage tool and ask it various questions about the environment, as follows: Checking OpenStack Compute Services To check our OpenStack Compute services, issue the following command: sudo nova-manage service list You will see an output similar to the following. The :-) indicates that everything is fine. nova-manage service list The fields are defined as follows: Binary: This is the name of the service that we're checking the status of. Host: This is name of the server or host where this service is running. Zone: This refers to the OpenStack Zone that is running that service. A zone can run different services. The default zone is called nova. Status: This states whether or not an administrator has enabled or disabled that service. State: This refers to whether that running service is working or not. Updated_At: This indicates when that service was last checked. If OpenStack Compute has a problem, you will see XXX in place of :-). The following command shows the same: nova-compute compute.book nova enabled XXX 2013-06-18 16:47:35 If you do see XXX, the answer to the problem will be in the logs at /var/log/nova/. If you get intermittent XXX and :-) for a service, first check whether the clocks are in sync. OpenStack Image Service (Glance) The OpenStack Image Service, Glance, while critical to the ability of OpenStack to provision new instances, does not contain its own tool to check the status of the service. Instead, we rely on some built-in Linux tools. OpenStack Image Service (Glance) doesn't have a tool to check its running services, so we can use some system commands instead, as follows: ps -ef | grep glance netstat -ant | grep 9292.*LISTEN These should return process information for Glance to show it's running, and 9292 is the default port that should be open in the LISTEN mode on your server, which is ready for use. The output of these commands will be similar to the following: ps -ef | grep glance This produces output like the following: To check if the correct port is in use, issue the following command: netstat -ant | grep 9292 tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN Other services that you should check Should Glance be having issues while the above services are in working order, you will want to check the following services as well: rabbitmq: For rabbitmq, run the following command: sudo rabbitmqctl status For example, output from rabbitmqctl (when everything is running OK) should look similar to the following screenshot: If rabbitmq isn't working as expected, you will see output similar to the following indicating that the rabbitmq service or node is down: ntp: For ntp (Network Time Protocol, for keeping nodes in time-sync), run the following command: ntpq -p ntp is required for multi-host OpenStack environments but it may not be installed by default. Install the ntp package with sudo apt-get install -y ntp) This should return output regarding contacting NTP servers, for example: MySQL Database Server: For MySQL Database Server, run the following commands: PASSWORD=openstack mysqladmin -uroot –p$PASSWORD status This will return some statistics about MySQL, if it is running, as shown in the following screenshot: Checking OpenStack Dashboard (Horizon) Like the Glance Service, the OpenStack Dashboard service, Horizon, does not come with a built-in tool to check its health. Horizon, despite not having a built-in utility to check service health, does rely on the Apache web server to serve pages. To check the status of the service then, we check the health of the web service. To check the health of the Apache web service, log into the server running Horizon and execute the following command: ps -ef | grep apache This command produces output like the following screenshot: To check that Apache is running on the expected port, TCP Port 80, issue the following command: netstat -ano | grep :80 This command should show the following output: tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN off (0.00/0/0) To test access to the web server from the command line issue the following command: telnet localhost 80 This command should show the following output: Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Checking OpenStack Identity (Keystone) Keystone comes with a client side implementation called the python-keystone client. We use this tool to check the status of our Keystone services. To check that Keystone is running the required services, we invoke the keystone command: # keystone user-list This produces output like the following screenshot: Additionally, you can use the following commands to check the status of Keystone. The following command checks the status of the service: # ps -ef | grep keystone This should show output similar to the following: keystone 5441 1 0 Jun20 ? 00:00:04 /usr/bin/python /usr/bin/keystone-all Next you can check that the service is listening on the network. The following command can be used: netstat -anlp | grep 5000 This command should show output like the following: tcp 0 0 0.0.0.0:5000 0.0.0.0: LISTEN 54421/python Checking OpenStack Networking (Neutron) When running the OpenStack Networking service, Neutron, there are a number of services that should be running on various nodes. These are depicted in the following diagram: On the Controller node, check the Quantum Server API service is running on TCP Port 9696 as follows: sudo netstat -anlp | grep 9696 The command brings back output like the following: tcp 0 0 0.0.0.0:9696 0.0.0.0:* LISTEN 22350/python On the Compute nodes, check the following services are running using the ps command: ovsdb-server ovs-switchd quantum-openvswitch-agent For example, run the following command: ps -ef | grep ovsdb-server On the Network node, check the following services are running: ovsdb-server ovs-switchd quantum-openvswitch-agent quantum-dhcp-agent quantum-l3-agent quantum-metadata-agent To check our Neutron agents are running correctly, issue the following command from the Controller host when you have the correct OpenStack credentials sourced into your environment: quantum agent-list This will bring back output like the following screenshot when everything is running correctly: Checking OpenStack Block Storage (Cinder) To check the status of the OpenStack Block Storage service, Cinder, you can use the following commands: Use the following command to check if Cinder is running: ps -ef | grep cinder This command produces output like the following screenshot: Use the following command to check if iSCSI target is listening: netstat -anp | grep 3260 This command produces output like the following: tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 10236/tgtd Use the following command to check that the Cinder API is listening on the network: netstat -an | grep 8776 This command produces output like the following: tcp 0 0.0.0.0:8776 0.0.0.0:* LISTEN To validate the operation of the Cinder service, if all of the above is functional, you can try to list the volumes Cinder knows about using the following: cinder list This produces output like the following: Checking OpenStack Object Storage (Swift) The OpenStack Object Storage service, Swift, has a few built-in utilities that allow us to check its health. To do so, log into your Swift node and run the following commands: Use the following command for checking the Swift Service Using Swift Stat: swift stat This produces output like the following: Using PS: There will be a service for each configured container, account, object-store. ps -ef | grep swift This should produce output like the following screenshot: Use the following command for checking the Swift API: ps -ef | grep swift-proxy This should produce the following screenshot: Use the following command for checking if Swift is listening on the network: netstat -anlp | grep 8080 This should produce output like the following: tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 9818/python How it works... We have used some basic commands that communicate with OpenStack services to show they're running. This elementary level of checking helps with troubleshooting our OpenStack environment. Troubleshooting OpenStack Compute services OpenStack Compute services are complex, and being able to diagnose faults is an essential part of ensuring the smooth running of the services. Fortunately, OpenStack Compute provides some tools to help with this process, along with tools provided by Ubuntu to help identify issues. How to do it... Troubleshooting OpenStack Compute services can be a complex issue, but working through problems methodically and logically will help you reach a satisfactory outcome. Carry out the following suggested steps when encountering the different problems presented. Steps for when you cannot ping or SSH to an instance When launching instances, we specify a security group. If none is specified, a security group named default is used. These mandatory security groups ensure security is enabled by default in our cloud environment, and as such, we must explicitly state that we require the ability to ping our instances and SSH to them. For such a basic activity, it is common to add these abilities to the default security group. Network issues may prevent us from accessing our cloud instances. First, check that the compute instances are able to forward packets from the public interface to the bridged interface. Use the following command for the same: sysctl -A | grep ip_forward net.ipv4.ip_forward should be set to 1. If it isn't, check that /etc/sysctl.conf has the following option uncommented. Use the following command for it: net.ipv4.ip_forward=1 Then, run to following command to pick up the change: sudo sysctl -p Other network issues could be routing issues. Check that we can communicate with the OpenStack Compute nodes from our client and that any routing to get to these instances has the correct entries. We may have a conflict with IPv6, if IPv6 isn't required. If this is the case, try adding --use_ipv6=false to your /etc/nova/nova.conf file, and restart the nova-compute and nova-network services. We may also need to disable IPv6 in the operating system, which can be achieved using something like the following line in /etc/modprobe.d/ipv6.conf: install ipv6 /bin/true If using OpenStack Neutron, check the status of the neutron services on the host and the correct IP namespace is being used (see Troubleshooting OpenStack Networking). Reboot your host. Methods for viewing the Instance Console log When using the command line, issue the following commands: nova list nova console-log INSTANCE_ID For example: nova console-log ee0cb5ca-281f-43e9-bb40-42ffddcb09cd When using Horizon, carry out the following steps: Navigate to the list of instance and select an instance. You will be taken to an Overview screen. Along the top of the Overview screen is a Log tab. This is the console log for the instance. When viewing the logs directly on a nova-compute host, look for the following file: The console logs are owned by root, so only an administrator can do this. They are placed at: var/lib/nova/instances/<instance_id>/console.log. Instance fails to download meta information If an instance fails to communicate to download the extra information that can be supplied to the instance meta-data, we can end up in a situation where the instance is up but you're unable to log in, as the SSH key information is injected using this method. Viewing the console log will show output like in the following screenshot: If you are not using Neutron, ensure the following: nova-api is running on the Controller host (in a multi_host environment, ensure there's a nova-api-metadata and a nova-network package installed and running on the Compute host). Perform the following iptables check on the Compute node: sudo iptables -L -n -t nat We should see a line in the output like in the following screenshot: If not, restart your nova-network services and check again. Sometimes there are multiple copies of dnsmasq running, which can cause this issue. Ensure that there is only one instance of dnsmasq running: ps -ef | grep dnsmasq This will bring back two process entries, the parent dnsmasq process and a spawned child (verify by the PIDs). If there are any other instances of dnsmasq running, kill the dnsmasq processes. When killed, restart nova-network, which will spawn dnsmasq again without any conflicting processes. If you are using Neutron: The first place to look is in the /var/log/quantum/metadata_agent.log on the Network host. Here you may see Python stack traces that could indicate a service isn't running correctly. A connection refused message may appear here suggesting the metadata agent running on the Network host is unable to talk to the Metadata service on the Controller host via the Metadata Proxy service (also running on the Network host). The metadata service runs on port 8775 on our Controller host, so checking that is running involves checking the port is open and it's running the metadata service. To do this on the Controller host, run the following: sudo netstat -antp | grep 8775 This will bring back the following output if everything is OK: tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN If nothing is returned, check that the nova-api service is running and if not, start it. Instance launches; stuck at Building or Pending Sometimes, a little patience is needed before assuming the instance has not booted, because the image is copied across the network to a node that has not seen the image before. At other times though, if the instance has been stuck in booting or a similar state for longer than normal, it indicates a problem. The first place to look will be for errors in the logs. A quick way of doing this is from the controller server and by issuing the following command: sudo nova-manage logs errors A common error that is usually present is usually related to AMQP being unreachable. Generally, these errors can be ignored unless, that is, you check the time stamp and these errors are currently appearing. You tend to see a number of these messages related to when the services first started up so look at the timestamp before reaching conclusions. This command brings back any log line with the ERROR as log level, but you will need to view the logs in more detail to get a clearer picture. A key log file, when troubleshooting instances that are not booting properly, will be available on the controller host at /var/log/nova/nova-scheduler.log. This file tends to produce the reason why an instance is stuck in Building state. Another file to view further information will be on the compute host at /var/log/nova/nova-compute.log. Look here at the time you launch the instance. In a busy environment, you will want to tail the log file and parse for the instance ID. Check /var/log/nova/nova-network.log (for Nova Network) and /var/log/quantum/*.log (for Neutron) for any reason why instances aren't being assigned IP addresses. It could be issues around DHCP preventing address allocation or quotas being reached. Error codes such as 401, 403, 500 The majority of the OpenStack services are web services, meaning the responses from the services are well defined. 40X: This refers to a service that is up but responding to an event that is produced by some user error. For example, a 401 is an authentication failure, so check the credentials used when accessing the service. 500: These errors mean a connecting service is unavailable or has caused an error that has caused the service to interpret a response to cause a failure. Common problems here are services that have not started properly, so check for running services. If all avenues have been exhausted when troubleshooting your environment, reach out to the community, using the mailing list or IRC, where there is a raft of people willing to offer their time and assistance. See the Getting help from the community recipe at the end of this article for more information. Listing all instances across all hosts From the OpenStack controller node, you can execute the following command to get a list of the running instances in the environment: sudo nova-manage vm list To view all instances across all tenants, as a user with an admin role execute the following command: nova list --all-tenants These commands are useful in identifying any failed instances and the host on which it is running. You can then investigate further. How it works... Troubleshooting OpenStack Compute problems can be quite complex, but looking in the right places can help solve some of the more common problems. Unfortunately, like troubleshooting any computer system, there isn't a single command that can help identify all the problems that you may encounter, but OpenStack provides some tools to help you identify some problems. Having an understanding of managing servers and networks will help troubleshoot a distributed cloud environment such as OpenStack. There's more than one place where you can go to identify the issues, as they can stem from the environment to the instances themselves. Methodically working your way through the problems though will help lead you to a resolution.

  HP Network Node Manager 9: Getting Started Manage your network effectively with NNMi Install, customize, and expand NNMi functionality by developing custom features Integrate NNMi with other management tools, such as HP SW Operations Manager, Network Automation, Cisco Works, Business Availability center, UCMDB, and many others Navigate between incidents and maps to reduce troubleshooting time Screenshots and step-by-step instructions to customize NNMi in the way you want Instructions in the book are valid for version 8 as well   Network Node Manager has a pretty long list of features, especially when we consider the information it provides in regards to network topology and all other information related to it. Every network is unique in terms of technologies it uses and purposes it is designed to. For example, carrying voice over IP, where voice converges with IP networks. MPLS is another unique technology, which in some terms can be treated as a separate science and needs additional management approach. Multicast is another story, with its own features and headaches from an operations perspective. All these technologies and features are not rocket science, but it is really an additional effort to be developed as a management tool. Most of NNMi users have hardly any of these technologies, so why should they pay for features they never use? Mostly, SPIs use NNMi's discovered nodes and their configuration as a primary source of information. Also, iSPI provides some information back to NNMi, that is, additional features or technology configuration, configuration changes, performance parameters, or alarms. Here is a list of major SPIs: iSPI for MPLS: Allows users to discover and monitor MPLS-specific objects and parameters. For example, L2/L3 VPNs, MVPN, Pseudo wire VC, VRF, PE-CE, PE-PE links, and so on. iSPI for IP Telephony: This iSPI discovers and monitors VoIP-specific objects and parameters. It supports VoIP monitoring from Avaya, Cisco, and Nortel vendors. iSPI Network Engineering toolset: This iSPI is a set of additional tools, which allows NNMi operator to initiate some routine actions, which helps in troubleshooting issues. iSPI for Performance: After NNMi version 8.11, this iSPI has been divided into two separate iSPIs: iSPI Performance for Metrics and iSPI Performance for Traffic. These iSPIs collect and report performance specific data—Network Engineering Toolset (NET). This iSPI provides additional troubleshooting and diagnostics tools for network engineers. iSPI for Multicast: This iSPI provides multicast network specific features, such as discovering and monitoring IP multicast routing topology, multicast enabled nodes, PIM interfaces and neighbors, and so on. Questions such as "Do I need SPI? If so, which one of these to choose? Will it do what I expect?" are ones commonly asked while designing NNMi. Let's take a look at the major SPIs. iSPI for Performance Before NNMi 8.11, there was iSPI Performance, which was introduced as two separate iSPIs on later NNMi versions. Legacy iSPI performance was collecting performance metrics based on SNMP queries on managed nodes. Later on, HP introduced the ability to collect data from flows, as flow data has a different list of features than performance data collected by SNMP. These iSPIs are: iSPI Performance for Metrics: Legacy iSPI performance with few improved features. iSPI Performance for Traffic: iSPI, which collects, analyzes, stores, and presents flow data. Let's take a look at both in detail. iSPI Performance for Metrics The iSPI Performance for Metrics adds the performance management capability to NNMi by analyzing, processing, and aggregating metrics collected by NNMi from different network elements. This release of the iSPI Performance for Metrics includes the following features: Path health reports omponent health reports Interface health reports Custom polled reports Also, unlike in previous NNM versions (7.x and earlier), we cannot trigger alerts based on performance data in basic NNMi versions. That is, we need to receive alarms when the device CPU load exceeds 95%, the interface utilization exceeds 70% or comes below 5% for longer than one hour. Default NNMi can't handle it. iSPI performance brings this feature into NNMi. Now, we can say that NNMi and iSPI Performance both together cover fault and performance monitoring areas. The network performance data adds more functionality for network management. It improves your network management by: Allowing operators to retrieve more data during investigations Enriching your monitoring by providing alerts based on performance data Providing information to network planners and analysts, where they can see long-term statistics, which makes future planning more accurate iSPI Performance collects, stores, arrays data, and presents it in drill-down reports. Using data mining in reports, we can drill down until we reach the node or interface, which causes issues. Users have relative flexibility in creating their own reports, as custom SQL queries can be created on reports by user-specific needs, such as a report with custom time period, or metrics, which are monitored. iSPI reports are reached from the NNM console, and no additional logins and passwords are needed as iSPI recognizes usernames or passwords used by NNMi. Reports running after they are selected and take up to 15 to 20 seconds. To eliminate this query at runtime, you can schedule the report to run in advance, as the scheduled report is displayed immediately. All monitored metrics can trigger threshold alarms, so operators can be notified before real impact occurs. Performance-based alarms also reflect the status of the nodes, which makes the map status more accurate for monitoring. Earlier NNM versions used to cause a lot of confusion when performance-based alarms indicated possible outages or upcoming service impacts, and map icons remained in a normal (green) state. NNMi iSPI Performance for Metrics may be licensed to monitor a smaller number of nodes than its corresponding NNMi. Consider, if you are a service provider and only a small part of your managed nodes has a requirement to be monitored features, which are supported by iSPI performance for metrics. Buying licenses for all the nodes would be a waste of money. Another wasteful example would be, if your NNMi, except routers and switches, also monitors a Users have relative flexibility in creating their own reports, large number of workstations or servers which don't need to be covered by iSPI Performance for Metrics. As HP changes licensing policy on a regular basis, please contact your HP representative to check the most current licensing policy. Please refer to http://support.openview.hp.com/selfsolve/manuals. NNMi can be configured to poll vendor proprietary MIBs, issue a threshold incident, set status on the map to alert operations, report on values and with NNMi iSPI performance for metrics. iSPI is not a replacement to HP Performance Insight (OVPI), but depending on particular requirements, sometimes iSPI Performance for Metrics may be used as an alternative to OVPI. The following table provides high-level comparison of iSPI Performance for Metrics and OVPI: NNMi iSPI Performance for Metrics Open View Performance Insight Tightly integrated with NNMi May be integrated into NNMi as well as can be a separate product Short to medium term data is stored Long-term data is stored Collects MIB data using SNMP Customized data collection methods can be used, that is, Operations Manager or Performance Manager agent Designed to be used for proactive monitoring and generating alarms Designed for long-term reporting Tool very handy for operations Tool for operations, analysis, planning, and reporting to management If you are, or plan to be an NNMi system administrator, you should be prepared to be asked whether or not iSPI Performance for Metrics loads a network with extra ICMP or SNMP traffic. Although the answer is yes, iSPI queries extra information, but on the other hand, it wouldn't load a network as much as it would separate a tool from a third party vendor, because iSPI uses the same SNMP process to collect performance and status information. It means that it eliminates extra polling, as the data is queried and responded using same packet bulks. iSPI Performance for Traffic By introducing this iSPI, NNMi took a step into network service monitoring. This iSPI uses flow data and can detect reports such as performance issues, like separate traffic types HTTP, mail, and FTP traffic. It can report about top sources or destinations, and so on. So now, the NNMi operator can take a look on what's happening inside IP traffic, as iSPI Performance for Traffic analyzes flow data. The following flow versions and vendors are supported: NetFlow: version 5, version 9 S-Flow: version 5 iSPI Performance for Traffic is very useful for troubleshooting network issues, such as: What kind of traffic utilizes my bandwidth most or fills it up? What sources or targets generate most traffic? These issues are a headache not only for operators, but for network or service analysts and planners as well. If your data channel is divided into traffic classes based on traffic type (that is, 20% of traffic for HTTP, 30% for mail, and so on), this iSPI will also tell you about your traffic classic behavior. Example: Why is my HTTP browsing is so slow, while my interface utilization is below 70%? Answer: HTTP traffic is configured to take max 30% of your bandwidth, and HTTP takes all of it while other traffic classes are less loaded, which makes total bandwidth utilization less than 100%. So, instead of constant questions and unclear situations, iSPI Performance for Traffic gives a clear answer, with evidence, that it is time to change traffic class allocation limits. This can be seen in the following diagrams: This iSPI can be used in conjunction with iSPI Performance for Metrics, which provides navigation between Metric and Traffic data. iSPI Performance for Traffic cannot trigger an alarm. Traffic generates performance reports from the IP flow records as follows: Aggregates the IP flow record. Correlates obtained IP flow records with NNMi topology for context-based analysis. Enriches the IP flow records by providing the ability to add or update the available fields in the flow records. For example, DNS name resolution and application mapping. Flow data is collected using flow collectors, which can be designed as two tier collectors: local and master. NNMi supports either of the two scenarios: co-located and non-co-located deployment of leaf, master collectors, and NNMi, as well as NNMi iSPI Performance server. The following figure represents the two-tier, hierarchical flow collection and processing: Leaf collectors: They are responsible for flow filtering, application mapping, DNS name resolution, and summary data feed to master collector. Master collector: This collector is responsible for collecting and correlating all summary records as central point from all leaf-level collectors. Common NNMi iSPI Performance Reporting Server: This server is responsible for building traffic analysis reports, which are done on the same server as iSPI Performance for Metrics. Multiple leaf collectors per physical machine can be supported. Can this iSPI be used as a replacement to OVPI? General answer is no. However, there may be some cases when iSPI Performance for Traffic may cover the required features. The following table provides a general comparison between iSPI and OVPI: iSPI Performance for Traffic Open View Performance Insight Tightly integrated with NNMi May be integrated into NNMi as well as can be separate product Short to medium term data is stored Long-term data is stored No alarms can be triggered Alarms can be triggered based on threshold settings Focused on flow collection Focused on long term trending, forecasting and capacity planning Tool, very handy for operations Tool for operations, analysis, planning and reporting to management Supports Net Flow (v5, v9) and sFlow (v5) Supports Net Flow (v5) and sFlow (v5, with IUM collector)  

CryENGINE 3 Cookbook Over 90 recipes written by Crytek developers for creating third-generation real-time games Low gravity In this simple recipe, we will look at utilizing the GravityBox to set up a low gravity area within a level. Getting ready Have Sandbox open Then open My_Level.cry How to do it... To start, first we must place down a GravityBox. In the RollupBar, click on the Entities button. Under the Physics section, select GravityBox. Place the GravityBox on the ground: Keeping the default dimensions (20, 20, 20 meters), the only property here that we want to change is the gravity. The default settings in this box set this entire area within the level to be a zero gravity zone. To adjust the up/down gravity of this, we need to change the value of gravity and the Z axis. To mimic normal gravity, this value would need to be set to the acceleration value of -9.81. To change this value to a lower gravity value, (something like the Moon's gravity) simply change it to a higher negative value such as -1.62. How it works... The GravityBox is a simple bounding box which overrides the defined gravity in the code (-9.81) and sets its own gravity value within the bounding box. Anything physicalized and activated to receive physics updates will behave within the confines of these gravitational rules unless they fall outside of the bounding box. There's more... Here are some useful tips about the gravity objects. Uniform property The uniform property within the GravityBox defines whether the GravityBox should use its own local orientation or the world's. If true, the GravityBox will use its own local rotation for the gravitational direction. If false, it will use the world's direction. This is used when you wish to have the gravity directed sideways. Set this value to True and then rotate the GravityBox onto its side. Gravity sphere Much like the GravityBox, the GravitySphere holds all the same principles but in a radius instead of a bounding box. The only other difference with the GravitySphere is that a false uniform Boolean will cause any object within the sphere to be attracted/repulsed from the center of the axis.   Hangman on a rope In this recipe, we will look at how we can utilize a rope to hang a dead body from it. Getting ready Open Sandbox Then open My_Level.cry How to do it... Begin by drawing out a rope: Open the RollupBar. From the Misc button, select Rope. With Grid Snap on and set to 1 meter, draw out a straight rope that has increments of one meter (by clicking once for every increment) up to four meters (double-click to finalize the rope). Align the rope so that from end to end it is along the Z axis (up and down) and a few meters off the ground: Next, we will need something solid to hang the rope from. Place down a solid with 1, 1, 1 meter. Align the rope underneath the solid cube while keeping both off the ground. Make sure when aligning the rope to get the end constraint to turn from red to green. This means it is attached to a physical surface: Lastly, we will need to hang a body from this rope. However, we will not hang him in the traditional manner, but rather by one of his feet. In the RollupBar, click on the Entities button. Under the Physics section, select DeadBody. Rotate this body up-side-down and align one of his feet to the bottom end of the rope. Select the rope to make sure the bottom constraint turns green to signal that it is attached. Verify that the Hangman on a rope recipe works by going into game mode and punching the dead body: How it works... The rope is a complicated cylinder that can contain as many bending segments as defined and is allowed to stretch and compress depending on the values defined. Tension and breaking strength can also be defined. But since ropes have expensive physics properties involved, they should be used sparingly.  

This article is written by Joe Kuan, the author of Learning Highcharts 4. All Highcharts graphs share the same configuration structure and it is crucial for us to become familiar with the core components. However, it is not possible to go through all the configurations within the book. In this article, we will explore the functional properties that are most used and demonstrate them with examples. We will learn how Highcharts manages layout, and then explore how to configure axes, specify single series and multiple series data, followed by looking at formatting and styling tool tips in both JavaScript and HTML. After that, we will get to know how to polish our charts with various types of animations and apply color gradients. Finally, we will explore the drilldown interactive feature. In this article, we will cover the following topics: Understanding Highcharts layout Framing the chart with axes (For more resources related to this topic, see here.) Configuration structure In the Highcharts configuration object, the components at the top level represent the skeleton structure of a chart. The following is a list of the major components that are covered in this article: chart: This has configurations for the top-level chart properties such as layouts, dimensions, events, animations, and user interactions series: This is an array of series objects (consisting of data and specific options) for single and multiple series, where the series data can be specified in a number of ways xAxis/yAxis/zAxis: This has configurations for all the axis properties such as labels, styles, range, intervals, plotlines, plot bands, and backgrounds tooltip: This has the layout and format style configurations for the series data tool tips drilldown: This has configurations for drilldown series and the ID field associated with the main series title/subtitle: This has the layout and style configurations for the chart title and subtitle legend: This has the layout and format style configurations for the chart legend plotOptions: This contains all the plotting options, such as display, animation, and user interactions, for common series and specific series types exporting: This has configurations that control the layout and the function of print and export features For reference information concerning all configurations, go to http://api.highcharts.com. Understanding Highcharts' layout Before we start to learn how Highcharts layout works, it is imperative that we understand some basic concepts first. First, set a border around the plot area. To do that we can set the options of plotBorderWidth and plotBorderColor in the chart section, as follows:         chart: {                renderTo: 'container',                type: 'spline',                plotBorderWidth: 1,                plotBorderColor: '#3F4044'        }, The second border is set around the Highcharts container. Next, we extend the preceding chart section with additional settings:         chart: {                renderTo: 'container',                ....                borderColor: '#a1a1a1',                borderWidth: 2,                borderRadius: 3        }, This sets the container border color with a width of 2 pixels and corner radius of 3 pixels. As we can see, there is a border around the container and this is the boundary that the Highcharts display cannot exceed: By default, Highcharts displays have three different areas: spacing, labeling, and plot area. The plot area is the area inside the inner rectangle that contains all the plot graphics. The labeling area is the area where labels such as title, subtitle, axis title, legend, and credits go, around the plot area, so that it is between the edge of the plot area and the inner edge of the spacing area. The spacing area is the area between the container border and the outer edge of the labeling area. The following screenshot shows three different kinds of areas. A gray dotted line is inserted to illustrate the boundary between the spacing and labeling areas. Each chart label position can be operated in one of the following two layouts: Automatic layout: Highcharts automatically adjusts the plot area size based on the labels' positions in the labeling area, so the plot area does not overlap with the label element at all. Automatic layout is the simplest way to configure, but has less control. This is the default way of positioning the chart elements. Fixed layout: There is no concept of labeling area. The chart label is specified in a fixed location so that it has a floating effect on the plot area. In other words, the plot area side does not automatically adjust itself to the adjacent label position. This gives the user full control of exactly how to display the chart. The spacing area controls the offset of the Highcharts display on each side. As long as the chart margins are not defined, increasing or decreasing the spacing area has a global effect on the plot area measurements in both automatic and fixed layouts. Chart margins and spacing settings In this section, we will see how chart margins and spacing settings have an effect on the overall layout. Chart margins can be configured with the properties margin, marginTop, marginLeft, marginRight, and marginBottom, and they are not enabled by default. Setting chart margins has a global effect on the plot area, so that none of the label positions or chart spacing configurations can affect the plot area size. Hence, all the chart elements are in a fixed layout mode with respect to the plot area. The margin option is an array of four margin values covered for each direction, the same as in CSS, starting from north and going clockwise. Also, the margin option has a lower precedence than any of the directional margin options, regardless of their order in the chart section. Spacing configurations are enabled by default with a fixed value on each side. These can be configured in the chart section with the property names spacing, spacingTop, spacingLeft, spacingBottom, and spacingRight. In this example, we are going to increase or decrease the margin or spacing property on each side of the chart and observe the effect. The following are the chart settings:             chart: {                renderTo: 'container',                type: ...                marginTop: 10,                marginRight: 0,                spacingLeft: 30,                spacingBottom: 0            }, The following screenshot shows what the chart looks like: The marginTop property fixes the plot area's top border 10 pixels away from the container border. It also changes the top border into fixed layout for any label elements, so the chart title and subtitle float on top of the plot area. The spacingLeft property increases the spacing area on the left-hand side, so it pushes the y axis title further in. As it is in automatic layout (without declaring marginLeft), it also pushes the plot area's west border in. Setting marginRight to 0 will override all the default spacing on the chart's right-hand side and change it to fixed layout mode. Finally, setting spacingBottom to 0 makes the legend touch the lower bar of the container, so it also stretches the plot area downwards. This is because the bottom edge is still in automatic layout even though spacingBottom is set to 0. Chart label properties Chart labels such as xAxis.title, yAxis.title, legend, title, subtitle, and credits share common property names, as follows: align: This is for the horizontal alignment of the label. Possible keywords are 'left', 'center', and 'right'. As for the axis title, it is 'low', 'middle', and 'high'. floating: This is to give the label position a floating effect on the plot area. Setting this to true will cause the label position to have no effect on the adjacent plot area's boundary. margin: This is the margin setting between the label and the side of the plot area adjacent to it. Only certain label types have this setting. verticalAlign: This is for the vertical alignment of the label. The keywords are 'top', 'middle', and 'bottom'. x: This is for horizontal positioning in relation to alignment. y: This is for vertical positioning in relation to alignment. As for the labels' x and y positioning, they are not used for absolute positioning within the chart. They are designed for fine adjustment with the label alignment. The following diagram shows the coordinate directions, where the center represents the label location: We can experiment with these properties with a simple example of the align and y position settings, by placing both title and subtitle next to each other. The title is shifted to the left with align set to 'left', whereas the subtitle alignment is set to 'right'. In order to make both titles appear on the same line, we change the subtitle's y position to 15, which is the same as the title's default y value:  title: {     text: 'Web browsers ...',     align: 'left' }, subtitle: {     text: 'From 2008 to present',     align: 'right',     y: 15 }, The following is a screenshot showing both titles aligned on the same line: In the following subsections, we will experiment with how changes in alignment for each label element affect the layout behavior of the plot area. Title and subtitle alignments Title and subtitle have the same layout properties, and the only differences are that the default values and title have the margin setting. Specifying verticalAlign for any value changes from the default automatic layout to fixed layout (it internally switches floating to true). However, manually setting the subtitle's floating property to false does not switch back to automatic layout. The following is an example of title in automatic layout and subtitle in fixed layout:     title: {       text: 'Web browsers statistics'    },    subtitle: {       text: 'From 2008 to present',       verticalAlign: 'top',       y: 60       }, The verticalAlign property for the subtitle is set to 'top', which switches the layout into fixed layout, and the y offset is increased to 60. The y offset pushes the subtitle's position further down. Due to the fact that the plot area is not in an automatic layout relationship to the subtitle anymore, the top border of the plot area goes above the subtitle. However, the plot area is still in automatic layout towards the title, so the title is still above the plot area: Legend alignment Legends show different behavior for the verticalAlign and align properties. Apart from setting the alignment to 'center', all other settings in verticalAlign and align remain in automatic positioning. The following is an example of a legend located on the right-hand side of the chart. The verticalAlign property is switched to the middle of the chart, where the horizontal align is set to 'right':           legend: {                align: 'right',                verticalAlign: 'middle',                layout: 'vertical'          }, The layout property is assigned to 'vertical' so that it causes the items inside the legend box to be displayed in a vertical manner. As we can see, the plot area is automatically resized for the legend box: Note that the border decoration around the legend box is disabled in the newer version. To display a round border around the legend box, we can add the borderWidth and borderRadius options using the following:           legend: {                align: 'right',                verticalAlign: 'middle',                layout: 'vertical',                borderWidth: 1,                borderRadius: 3          }, Here is the legend box with a round corner border: Axis title alignment Axis titles do not use verticalAlign. Instead, they use the align setting, which is either 'low', 'middle', or 'high'. The title's margin value is the distance between the axis title and the axis line. The following is an example of showing the y-axis title rotated horizontally instead of vertically (which it is by default) and displayed on the top of the axis line instead of next to it. We also use the y property to fine-tune the title location:             yAxis: {                title: {                    text: 'Percentage %',                    rotation: 0,                    y: -15,                    margin: -70,                    align: 'high'                },                min: 0            }, The following is a screenshot of the upper-left corner of the chart showing that the title is aligned horizontally at the top of the y axis. Alternatively, we can use the offset option instead of margin to achieve the same result. Credits alignment Credits is a bit different from other label elements. It only supports the align, verticalAlign, x, and y properties in the credits.position property (shorthand for credits: { position: … }), and is also not affected by any spacing setting. Suppose we have a graph without a legend and we have to move the credits to the lower-left area of the chart, the following code snippet shows how to do it:             legend: {                enabled: false            },            credits: {                position: {                   align: 'left'                },                text: 'Joe Kuan',                href: 'http://joekuan.wordpress.com'            }, However, the credits text is off the edge of the chart, as shown in the following screenshot: Even if we move the credits label to the right with x positioning, the label is still a bit too close to the x axis interval label. We can introduce extra spacingBottom to put a gap between both labels, as follows:             chart: {                   spacingBottom: 30,                    ....            },            credits: {                position: {                   align: 'left',                   x: 20,                   y: -7                },            },            .... The following is a screenshot of the credits with the final adjustments: Experimenting with an automatic layout In this section, we will examine the automatic layout feature in more detail. For the sake of simplifying the example, we will start with only the chart title and without any chart spacing settings:      chart: {         renderTo: 'container',         // border and plotBorder settings         borderWidth: 2,         .....     },     title: {            text: 'Web browsers statistics,     }, From the preceding example, the chart title should appear as expected between the container and the plot area's borders: The space between the title and the top border of the container has the default setting spacingTop for the spacing area (a default value of 10-pixels high). The gap between the title and the top border of the plot area is the default setting for title.margin, which is 15-pixels high. By setting spacingTop in the chart section to 0, the chart title moves up next to the container top border. Hence the size of the plot area is automatically expanded upwards, as follows: Then, we set title.margin to 0; the plot area border moves further up, hence the height of the plot area increases further, as follows: As you may notice, there is still a gap of a few pixels between the top border and the chart title. This is actually due to the default value of the title's y position setting, which is 15 pixels, large enough for the default title font size. The following is the chart configuration for setting all the spaces between the container and the plot area to 0: chart: {     renderTo: 'container',     // border and plotBorder settings     .....     spacingTop: 0},title: {     text: null,     margin: 0,     y: 0} If we set title.y to 0, all the gap between the top edge of the plot area and the top container edge closes up. The following is the final screenshot of the upper-left corner of the chart, to show the effect. The chart title is not visible anymore as it has been shifted above the container: Interestingly, if we work backwards to the first example, the default distance between the top of the plot area and the top of the container is calculated as: spacingTop + title.margin + title.y = 10 + 15 + 15 = 40 Therefore, changing any of these three variables will automatically adjust the plot area from the top container bar. Each of these offset variables actually has its own purpose in the automatic layout. Spacing is for the gap between the container and the chart content; thus, if we want to display a chart nicely spaced with other elements on a web page, spacing elements should be used. Equally, if we want to use a specific font size for the label elements, we should consider adjusting the y offset. Hence, the labels are still maintained at a distance and do not interfere with other components in the chart. Experimenting with a fixed layout In the preceding section, we have learned how the plot area dynamically adjusted itself. In this section, we will see how we can manually position the chart labels. First, we will start with the example code from the beginning of the Experimenting with automatic layout section and set the chart title's verticalAlign to 'bottom', as follows: chart: {    renderTo: 'container',    // border and plotBorder settings    .....},title: {    text: 'Web browsers statistics',    verticalAlign: 'bottom'}, The chart title is moved to the bottom of the chart, next to the lower border of the container. Notice that this setting has changed the title into floating mode; more importantly, the legend still remains in the default automatic layout of the plot area: Be aware that we haven't specified spacingBottom, which has a default value of 15 pixels in height when applied to the chart. This means that there should be a gap between the title and the container bottom border, but none is shown. This is because the title.y position has a default value of 15 pixels in relation to spacing. According to the diagram in the Chart label properties section, this positive y value pushes the title towards the bottom border; this compensates for the space created by spacingBottom. Let's make a bigger change to the y offset position this time to show that verticalAlign is floating on top of the plot area:  title: {     text: 'Web browsers statistics',     verticalAlign: 'bottom',     y: -90 }, The negative y value moves the title up, as shown here: Now the title is overlapping the plot area. To demonstrate that the legend is still in automatic layout with regard to the plot area, here we change the legend's y position and the margin settings, which is the distance from the axis label:                legend: {                   margin: 70,                   y: -10               }, This has pushed up the bottom side of the plot area. However, the chart title still remains in fixed layout and its position within the chart hasn't been changed at all after applying the new legend setting, as shown in the following screenshot: By now, we should have a better understanding of how to position label elements, and their layout policy relating to the plot area. Framing the chart with axes In this section, we are going to look into the configuration of axes in Highcharts in terms of their functional area. We will start off with a plain line graph and gradually apply more options to the chart to demonstrate the effects. Accessing the axis data type There are two ways to specify data for a chart: categories and series data. For displaying intervals with specific names, we should use the categories field that expects an array of strings. Each entry in the categories array is then associated with the series data array. Alternatively, the axis interval values are embedded inside the series data array. Then, Highcharts extracts the series data for both axes, interprets the data type, and formats and labels the values appropriately. The following is a straightforward example showing the use of categories:     chart: {        renderTo: 'container',        height: 250,        spacingRight: 20    },    title: {        text: 'Market Data: Nasdaq 100'    },    subtitle: {        text: 'May 11, 2012'    },    xAxis: {        categories: [ '9:30 am', '10:00 am', '10:30 am',                       '11:00 am', '11:30 am', '12:00 pm',                       '12:30 pm', '1:00 pm', '1:30 pm',                       '2:00 pm', '2:30 pm', '3:00 pm',                       '3:30 pm', '4:00 pm' ],         labels: {             step: 3         }     },     yAxis: {         title: {             text: null         }     },     legend: {         enabled: false     },     credits: {         enabled: false     },     series: [{         name: 'Nasdaq',         color: '#4572A7',         data: [ 2606.01, 2622.08, 2636.03, 2637.78, 2639.15,                 2637.09, 2633.38, 2632.23, 2632.33, 2632.59,                 2630.34, 2626.89, 2624.59, 2615.98 ]     }] The preceding code snippet produces a graph that looks like the following screenshot: The first name in the categories field corresponds to the first value, 9:30 am, 2606.01, in the series data array, and so on. Alternatively, we can specify the time values inside the series data and use the type property of the x axis to format the time. The type property supports 'linear' (default), 'logarithmic', or 'datetime'. The 'datetime' setting automatically interprets the time in the series data into human-readable form. Moreover, we can use the dateTimeLabelFormats property to predefine the custom format for the time unit. The option can also accept multiple time unit formats. This is for when we don't know in advance how long the time span is in the series data, so each unit in the resulting graph can be per hour, per day, and so on. The following example shows how the graph is specified with predefined hourly and minute formats. The syntax of the format string is based on the PHP strftime function:     xAxis: {         type: 'datetime',          // Format 24 hour time to AM/PM          dateTimeLabelFormats: {                hour: '%I:%M %P',              minute: '%I %M'          }               },     series: [{         name: 'Nasdaq',         color: '#4572A7',         data: [ [ Date.UTC(2012, 4, 11, 9, 30), 2606.01 ],                  [ Date.UTC(2012, 4, 11, 10), 2622.08 ],                   [ Date.UTC(2012, 4, 11, 10, 30), 2636.03 ],                  .....                ]     }] Note that the x axis is in the 12-hour time format, as shown in the following screenshot: Instead, we can define the format handler for the xAxis.labels.formatter property to achieve a similar effect. Highcharts provides a utility routine, Highcharts.dateFormat, that converts the timestamp in milliseconds to a readable format. In the following code snippet, we define the formatter function using dateFormat and this.value. The keyword this is the axis's interval object, whereas this.value is the UTC time value for the instance of the interval:     xAxis: {         type: 'datetime',         labels: {             formatter: function() {                 return Highcharts.dateFormat('%I:%M %P', this.value);             }         }     }, Since the time values of our data points are in fixed intervals, they can also be arranged in a cut-down version. All we need is to define the starting point of time, pointStart, and the regular interval between them, pointInterval, in milliseconds: series: [{     name: 'Nasdaq',     color: '#4572A7',     pointStart: Date.UTC(2012, 4, 11, 9, 30),     pointInterval: 30 * 60 * 1000,     data: [ 2606.01, 2622.08, 2636.03, 2637.78,             2639.15, 2637.09, 2633.38, 2632.23,             2632.33, 2632.59, 2630.34, 2626.89,             2624.59, 2615.98 ] }] Adjusting intervals and background We have learned how to use axis categories and series data arrays in the last section. In this section, we will see how to format interval lines and the background style to produce a graph with more clarity. We will continue from the previous example. First, let's create some interval lines along the y axis. In the chart, the interval is automatically set to 20. However, it would be clearer to double the number of interval lines. To do that, simply assign the tickInterval value to 10. Then, we use minorTickInterval to put another line in between the intervals to indicate a semi-interval. In order to distinguish between interval and semi-interval lines, we set the semi-interval lines, minorGridLineDashStyle, to a dashed and dotted style. There are nearly a dozen line style settings available in Highcharts, from 'Solid' to 'LongDashDotDot'. Readers can refer to the online manual for possible values. The following is the first step to create the new settings:             yAxis: {                 title: {                     text: null                 },                 tickInterval: 10,                 minorTickInterval: 5,                 minorGridLineColor: '#ADADAD',                 minorGridLineDashStyle: 'dashdot'            } The interval lines should look like the following screenshot: To make the graph even more presentable, we add a striping effect with shading using alternateGridColor. Then, we change the interval line color, gridLineColor, to a similar range with the stripes. The following code snippet is added into the yAxis configuration:                 gridLineColor: '#8AB8E6',                 alternateGridColor: {                     linearGradient: {                         x1: 0, y1: 1,                         x2: 1, y2: 1                     },                     stops: [ [0, '#FAFCFF' ],                              [0.5, '#F5FAFF'] ,                              [0.8, '#E0F0FF'] ,                              [1, '#D6EBFF'] ]                   } The following is the graph with the new shading background: The next step is to apply a more professional look to the y axis line. We are going to draw a line on the y axis with the lineWidth property, and add some measurement marks along the interval lines with the following code snippet:                  lineWidth: 2,                  lineColor: '#92A8CD',                  tickWidth: 3,                  tickLength: 6,                  tickColor: '#92A8CD',                  minorTickLength: 3,                  minorTickWidth: 1,                  minorTickColor: '#D8D8D8' The tickWidth and tickLength properties add the effect of little marks at the start of each interval line. We apply the same color on both the interval mark and the axis line. Then we add the ticks minorTickLength and minorTickWidth into the semi-interval lines in a smaller size. This gives a nice measurement mark effect along the axis, as shown in the following screenshot: Now, we apply a similar polish to the xAxis configuration, as follows:            xAxis: {                type: 'datetime',                labels: {                    formatter: function() {                        return Highcharts.dateFormat('%I:%M %P', this.value);                    },                },                gridLineDashStyle: 'dot',                gridLineWidth: 1,                tickInterval: 60 * 60 * 1000,                lineWidth: 2,                lineColor: '#92A8CD',                tickWidth: 3,                tickLength: 6,                tickColor: '#92A8CD',            }, We set the x axis interval lines to the hourly format and switch the line style to a dotted line. Then, we apply the same color, thickness, and interval ticks as on the y axis. The following is the resulting screenshot: However, there are some defects along the x axis line. To begin with, the meeting point between the x axis and y axis lines does not align properly. Secondly, the interval labels at the x axis are touching the interval ticks. Finally, part of the first data point is covered by the y-axis line. The following is an enlarged screenshot showing the issues: There are two ways to resolve the axis line alignment problem, as follows: Shift the plot area 1 pixel away from the x axis. This can be achieved by setting the offset property of xAxis to 1. Increase the x-axis line width to 3 pixels, which is the same width as the y-axis tick interval. As for the x-axis label, we can simply solve the problem by introducing the y offset value into the labels setting. Finally, to avoid the first data point touching the y-axis line, we can impose minPadding on the x axis. What this does is to add padding space at the minimum value of the axis, the first point. The minPadding value is based on the ratio of the graph width. In this case, setting the property to 0.02 is equivalent to shifting along the x axis 5 pixels to the right (250 px * 0.02). The following are the additional settings to improve the chart:     xAxis: {         ....         labels: {                formatter: ...,                y: 17         },         .....         minPadding: 0.02,         offset: 1     } The following screenshot shows that the issues have been addressed: As we can see, Highcharts has a comprehensive set of configurable variables with great flexibility. Using plot lines and plot bands In this section, we are going to see how we can use Highcharts to place lines or bands along the axis. We will continue with the example from the previous section. Let's draw a couple of lines to indicate the day's highest and lowest index points on the y axis. The plotLines field accepts an array of object configurations for each plot line. There are no width and color default values for plotLines, so we need to specify them explicitly in order to see the line. The following is the code snippet for the plot lines:       yAxis: {               ... ,               plotLines: [{                    value: 2606.01,                    width: 2,                    color: '#821740',                    label: {                        text: 'Lowest: 2606.01',                        style: {                            color: '#898989'                        }                    }               }, {                    value: 2639.15,                    width: 2,                    color: '#4A9338',                    label: {                        text: 'Highest: 2639.15',                        style: {                            color: '#898989'                        }                    }               }]         } The following screenshot shows what it should look like: We can improve the look of the chart slightly. First, the text label for the top plot line should not be next to the highest point. Second, the label for the bottom line should be remotely covered by the series and interval lines, as follows: To resolve these issues, we can assign the plot line's zIndex to 1, which brings the text label above the interval lines. We also set the x position of the label to shift the text next to the point. The following are the new changes:              plotLines: [{                    ... ,                    label: {                        ... ,                        x: 25                    },                    zIndex: 1                    }, {                    ... ,                    label: {                        ... ,                        x: 130                    },                    zIndex: 1               }] The following graph shows the label has been moved away from the plot line and over the interval line: Now, we are going to change the preceding example with a plot band area that shows the index change between the market's opening and closing values. The plot band configuration is very similar to plot lines, except that it uses the to and from properties, and the color property accepts gradient settings or color code. We create a plot band with a triangle text symbol and values to signify a positive close. Instead of using the x and y properties to fine-tune label position, we use the align option to adjust the text to the center of the plot area (replace the plotLines setting from the above example):               plotBands: [{                    from: 2606.01,                    to: 2615.98,                    label: {                        text: '▲ 9.97 (0.38%)',                        align: 'center',                        style: {                            color: '#007A3D'                        }                    },                    zIndex: 1,                    color: {                        linearGradient: {                            x1: 0, y1: 1,                            x2: 1, y2: 1                        },                        stops: [ [0, '#EBFAEB' ],                                 [0.5, '#C2F0C2'] ,                                 [0.8, '#ADEBAD'] ,                                 [1, '#99E699']                        ]                    }               }] The triangle is an alt-code character; hold down the left Alt key and enter 30 in the number keypad. See http://www.alt-codes.net for more details. This produces a chart with a green plot band highlighting a positive close in the market, as shown in the following screenshot: Extending to multiple axes Previously, we ran through most of the axis configurations. Here, we explore how we can use multiple axes, which are just an array of objects containing axis configurations. Continuing from the previous stock market example, suppose we now want to include another market index, Dow Jones, along with Nasdaq. However, both indices are different in nature, so their value ranges are vastly different. First, let's examine the outcome by displaying both indices with the common y axis. We change the title, remove the fixed interval setting on the y axis, and include data for another series:             chart: ... ,             title: {                 text: 'Market Data: Nasdaq & Dow Jones'             },             subtitle: ... ,             xAxis: ... ,             credits: ... ,             yAxis: {                 title: {                     text: null                 },                 minorGridLineColor: '#D8D8D8',                 minorGridLineDashStyle: 'dashdot',                 gridLineColor: '#8AB8E6',                 alternateGridColor: {                     linearGradient: {                         x1: 0, y1: 1,                         x2: 1, y2: 1                     },                     stops: [ [0, '#FAFCFF' ],                              [0.5, '#F5FAFF'] ,                              [0.8, '#E0F0FF'] ,                              [1, '#D6EBFF'] ]                 },                 lineWidth: 2,                 lineColor: '#92A8CD',                 tickWidth: 3,                 tickLength: 6,                 tickColor: '#92A8CD',                 minorTickLength: 3,                 minorTickWidth: 1,                 minorTickColor: '#D8D8D8'             },             series: [{               name: 'Nasdaq',               color: '#4572A7',               data: [ [ Date.UTC(2012, 4, 11, 9, 30), 2606.01 ],                          [ Date.UTC(2012, 4, 11, 10), 2622.08 ],                           [ Date.UTC(2012, 4, 11, 10, 30), 2636.03 ],                          ...                        ]             }, {               name: 'Dow Jones',               color: '#AA4643',               data: [ [ Date.UTC(2012, 4, 11, 9, 30), 12598.32 ],                          [ Date.UTC(2012, 4, 11, 10), 12538.61 ],                           [ Date.UTC(2012, 4, 11, 10, 30), 12549.89 ],                          ...                        ]             }] The following is the chart showing both market indices: As expected, the index changes that occur during the day have been normalized by the vast differences in value. Both lines look roughly straight, which falsely implies that the indices have hardly changed. Let us now explore putting both indices onto separate y axes. We should remove any background decoration on the y axis, because we now have a different range of data shared on the same background. The following is the new setup for yAxis:            yAxis: [{                  title: {                     text: 'Nasdaq'                 },               }, {                 title: {                     text: 'Dow Jones'                 },                 opposite: true             }], Now yAxis is an array of axis configurations. The first entry in the array is for Nasdaq and the second is for Dow Jones. This time, we display the axis title to distinguish between them. The opposite property is to put the Dow Jones y axis onto the other side of the graph for clarity. Otherwise, both y axes appear on the left-hand side. The next step is to align indices from the y-axis array to the series data array, as follows:             series: [{                 name: 'Nasdaq',                 color: '#4572A7',                 yAxis: 0,                 data: [ ... ]             }, {                 name: 'Dow Jones',                 color: '#AA4643',                 yAxis: 1,                 data: [ ... ]             }]          We can clearly see the movement of the indices in the new graph, as follows: Moreover, we can improve the final view by color-matching the series to the axis lines. The Highcharts.getOptions().colors property contains a list of default colors for the series, so we use the first two entries for our indices. Another improvement is to set maxPadding for the x axis, because the new y-axis line covers parts of the data points at the high end of the x axis:             xAxis: {                 ... ,                 minPadding: 0.02,                 maxPadding: 0.02                 },             yAxis: [{                 title: {                     text: 'Nasdaq'                 },                 lineWidth: 2,                 lineColor: '#4572A7',                 tickWidth: 3,                 tickLength: 6,                 tickColor: '#4572A7'             }, {                 title: {                     text: 'Dow Jones'                 },                 opposite: true,                 lineWidth: 2,                 lineColor: '#AA4643',                 tickWidth: 3,                 tickLength: 6,                 tickColor: '#AA4643'             }], The following screenshot shows the improved look of the chart: We can extend the preceding example and have more than a couple of axes, simply by adding entries into the yAxis and series arrays, and mapping both together. The following screenshot shows a 4-axis line graph: Summary In this article, major configuration components were discussed and experimented with, and examples shown. By now, we should be comfortable with what we have covered already and ready to plot some of the basic graphs with more elaborate styles. Resources for Article: Further resources on this subject: Theming with Highcharts [article] Integrating with other Frameworks [article] Highcharts [article]

 In this article by Giancarlo Zaccone, the author of the book Python Parallel Programming Cookbook, we will cover the following topics: Introducing Asyncio GPU programming with Python Introducing PyCUDA Introducing PyOpenCL (For more resources related to this topic, see here.) An asynchronous model is of fundamental importance along with the concept of event programming. The execution model of asynchronous activities can be implemented using a single stream of main control, both in uniprocessor systems and multiprocessor systems. In the asynchronous model of a concurrent execution, various tasks intersect with each other along the timeline, and all of this happens under the action of a single flow of control (single-threaded). The execution of a task can be suspended and then resumed alternating in time with any other task. The asynchronous programming model As you can see in the preceding figure, the tasks (each with a different color) are interleaved with one another, but they are in a single thread of control. This implies that when one task is in execution, the other tasks are not. A key difference between a multithreaded programming model and single-threaded asynchronous concurrent model is that in the first case, the operating system decides on the timeline whether to suspend the activity of a thread and start another, while in the second case, the programmer must assume that a thread may be suspended and replaced with another at almost any time. Introducing Asyncio The Python module Asyncio provides facilities to manage events, coroutines, tasks and threads, and synchronization primitives to write concurrent code. When a program becomes very long and complex, it is convenient to divide it into subroutines, each of which realizes a specific task, for which the program implements a suitable algorithm. The subroutine cannot be executed independently but only at the request of the main program, which is then responsible for coordinating the use of subroutines. Coroutines are a generalization of the subroutine. Like a subroutine, the coroutine computes a single computational step, but unlike subroutines, there is no main program that is used to coordinate the results. This is because the coroutines link themselves together to form a pipeline without any supervising function responsible for calling them in a particular order. In a coroutine, the execution point can be suspended and resumed later, having kept track of its local state in the intervening time. In this example, we see how to use the coroutine mechanism of Asyncio to simulate a finite state machine of five states. A Finite-state automaton (FSA) is a mathematical model that is not only widely used in engineering disciplines but also in sciences, such as mathematics and computer science. The automata we want to simulate the behavior is as follows: Finite State Machine We have indicated with S0, S1, S2, S3, and S4 the states of the system, with 0 and 1 as the values for which the automata can pass from one state to the next state (this operation is called a transition). So for example, the state S0 can be passed to the state S1 only for the value 1 and S0 can pass the state S2 only to the value 0. The Python code that follows simulates a transition of the automaton from the state S0, the so-called Start State, up to the state S4, the End State: #Asyncio Finite State Machine import asyncio import time from random import randint @asyncio.coroutine def StartState(): print ("Start State called n") input_value = randint(0,1) time.sleep(1) if (input_value == 0): result = yield from State2(input_value) else : result = yield from State1(input_value) print("Resume of the Transition : nStart State calling " + result) @asyncio.coroutine def State1(transition_value): outputValue = str(("State 1 with transition value = %s n" %(transition_value))) input_value = randint(0,1) time.sleep(1) print("...Evaluating...") if (input_value == 0): result = yield from State3(input_value) else : result = yield from State2(input_value) result = "State 1 calling " + result return (outputValue + str(result)) @asyncio.coroutine def State2(transition_value): outputValue = str(("State 2 with transition value = %s n" %(transition_value))) input_value = randint(0,1) time.sleep(1) print("...Evaluating...") if (input_value == 0): result = yield from State1(input_value) else : result = yield from State3(input_value) result = "State 2 calling " + result return (outputValue + str(result)) @asyncio.coroutine def State3(transition_value): outputValue = str(("State 3 with transition value = %s n" %(transition_value))) input_value = randint(0,1) time.sleep(1) print("...Evaluating...") if (input_value == 0): result = yield from State1(input_value) else : result = yield from EndState(input_value) result = "State 3 calling " + result return (outputValue + str(result)) @asyncio.coroutine def EndState(transition_value): outputValue = str(("End State with transition value = %s n" %(transition_value))) print("...Stop Computation...") return (outputValue ) if __name__ == "__main__": print("Finite State Machine simulation with Asyncio Coroutine") loop = asyncio.get_event_loop() loop.run_until_complete(StartState()) After running the code, we have an output similar to this: C:Python CookBookChapter 4- Asynchronous Programmingcodes - Chapter 4>python asyncio_state_machine.py Finite State Machine simulation with Asyncio Coroutine Start State called ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Evaluating... ...Stop Computation... Resume of the Transition : Start State calling State 1 with transition value = 1 State 1 calling State 3 with transition value = 0 State 3 calling State 1 with transition value = 0 State 1 calling State 2 with transition value = 1 State 2 calling State 3 with transition value = 1 State 3 calling State 1 with transition value = 0 State 1 calling State 2 with transition value = 1 State 2 calling State 1 with transition value = 0 State 1 calling State 3 with transition value = 0 State 3 calling State 1 with transition value = 0 State 1 calling State 2 with transition value = 1 State 2 calling State 3 with transition value = 1 State 3 calling End State with transition value = 1 Each state of the automata has been defined with the annotation @asyncio.coroutine. For example, the state S0 is: @asyncio.coroutine def StartState(): print ("Start State called n") input_value = randint(0,1) time.sleep(1) if (input_value == 0): result = yield from State2(input_value) else : result = yield from State1(input_value) The transition to the next state is determined by input_value, which is defined by the randint(0,1) function of Python's module random. This function randomly provides the value 0 or 1, where it randomly determines to which state the finite-state machine will be passed: input_value = randint(0,1) After determining the value at which state the finite state machine will be passed, the coroutine calls the next coroutine using the command yield from: if (input_value == 0): result = yield from State2(input_value) else : result = yield from State1(input_value) The variable result is the value that each coroutine returns. It is a string, and at the end of the computation, we can reconstruct [NV1] the transition from the initial state of the automaton, the Start State, up to the final state, the End State. The main program starts the evaluation inside the event loop: if __name__ == "__main__": print("Finite State Machine simulation with Asyncio Coroutine") loop = asyncio.get_event_loop() loop.run_until_complete(StartState()) GPU programming with Python A graphics processing unit (GPU) is an electronic circuit that specializes in processing data to render images from polygonal primitives. Although they were designed to carry out rendering images, GPUs have continued to evolve, becoming more complex and efficient in serving both real-time and offline rendering community. GPUs have continued to evolve, becoming more complex and efficient in performing any scientific computation. Each GPU is indeed composed of several processing units called streaming multiprocessor (SM), representing the first logic level of parallelism; each SM in fact, works simultaneously and independently from the others. The GPU architecture Each SM is in turn divided into a group of Stream Processors (SP), each of which has a core of real execution and can run a thread sequentially. SP represents the smallest unit of execution logic and the level of finer parallelism. The division in SM and SP is structural in nature, but it is possible to outline a further logical organization of the SP of a GPU, which are grouped together in logical blocks characterized by a particular mode of execution—all cores that make up a group run at the same time with the same instructions. This is just the SIMD (Single Instruction, Multiple Data) model. The programming paradigm that characterizes GPU computing is also called stream processing because the data can be viewed as a homogeneous flow of values that are applied synchronously to the same operations. Currently, the most efficient solutions to exploit the computing power provided by the GPU cards are the software libraries CUDA and OpenCL. Introducing PyCUDA PyCUDA is a Python wrapper for CUDA (Compute Unified Device Architecture), the software library developed by NVIDIA for GPU programming. The PyCuda programming model is designed for the common execution of a program on the CPU and GPU so as to allow you to perform the sequential parts on the CPU and the numeric parts that are more intensive on the GPU. The phases to be performed in the sequential mode are implemented and executed on the CPU (host), while the steps to be performed in parallel are implemented and executed on the GPU (device). The functions to be performed in parallel on the device are called kernels. The skeleton general for the execution of a generic function kernel on the device is as follows: Allocation of memory on the device. Transfer of data from the host memory to that allocated on the device. Running the device: Running the configuration. Invocation of the kernel function. Transfer of the results from the memory on the device to the host memory. Release of the memory allocated on the device. The PyCUDA programming model To show the PyCuda workflow, let's consider a 5 × 5 random array and the following procedure: Create the array 5×5 on the CPU. Transfer the array to the GPU. Perform a Task[NV2]  on the array in the GPU (double all the items in the array). Transfer the array from the GPU to the CPU. Print the results. The code for this is as follows: import pycuda.driver as cuda import pycuda.autoinit from pycuda.compiler import SourceModule import numpy a = numpy.random.randn(5,5) a = a.astype(numpy.float32) a_gpu = cuda.mem_alloc(a.nbytes) cuda.memcpy_htod(a_gpu, a) mod = SourceModule(""" __global__ void doubleMatrix(float *a) { int idx = threadIdx.x + threadIdx.y*4; a[idx] *= 2; } """) func = mod.get_function("doubleMatrix") func(a_gpu, block=(5,5,1)) a_doubled = numpy.empty_like(a) cuda.memcpy_dtoh(a_doubled, a_gpu) print ("ORIGINAL MATRIX") print a print ("DOUBLED MATRIX AFTER PyCUDA EXECUTION") print a_doubled The example output should be like this : C:Python CookBookChapter 6 - GPU Programming with Python >python PyCudaWorkflow.py ORIGINAL MATRIX [[-0.59975582 1.93627465 0.65337795 0.13205571 -0.46468592] [ 0.01441949 1.40946579 0.5343408 -0.46614054 -0.31727529] [-0.06868593 1.21149373 -0.6035406 -1.29117763 0.47762445] [ 0.36176383 -1.443097 1.21592784 -1.04906416 -1.18935871] [-0.06960868 -1.44647694 -1.22041082 1.17092752 0.3686313 ]] DOUBLED MATRIX AFTER PyCUDA EXECUTION [[-1.19951165 3.8725493 1.3067559 0.26411143 -0.92937183] [ 0.02883899 2.81893158 1.0686816 -0.93228108 -0.63455057] [-0.13737187 2.42298746 -1.2070812 -2.58235526 0.95524889] [ 0.72352767 -1.443097 1.21592784 -1.04906416 -1.18935871] [-0.06960868 -1.44647694 -1.22041082 1.17092752 0.3686313 ]] The code starts with the following imports: import pycuda.driver as cuda import pycuda.autoinit from pycuda.compiler import SourceModule The pycuda.autoinit import automatically picks a GPU to run on based on the availability and number. It also creates a GPU context for subsequent code to run in. Both the chosen device and the created context are available from pycuda.autoinit as importable symbols if needed. While the SourceModule component is the object where a C-like code for the GPU must be written. The first step is to generate the input 5 × 5 matrix. Since most GPU computations involve large arrays of data, the NumPy module must be imported: import numpy a = numpy.random.randn(5,5) Then, the items in the matrix are converted in a single precision mode, many NVIDIA cards support only single precision: a = a.astype(numpy.float32) The first operation to be done in order to implement a GPU loads the input array from the host memory (CPU) to the device (GPU). This is done at the beginning of the operation and consists two steps that are performed by invoking two functions provided PyCuda[NV3] . Memory allocation on the device is done via the cuda.mem_alloc function. The device and host memory may not ever communicate while performing a function kernel. This means that to run a function in parallel on the device, the data relating to it must be present in the memory of the device itself. Before you copy data from the host memory to the device memory, you must allocate the memory required on the device: a_gpu = cuda.mem_alloc(a.nbytes). Copy of the matrix from the host memory to that of the device with the function: call cuda.memcpy_htod : cuda.memcpy_htod(a_gpu, a). We also note that a_gpu is one dimensional, and on the device, we need to handle it as such. All these operations do not require the invocation of a kernel and are made directly by the main processor. The SourceModule entity serves to define the (C-like) kernel function doubleMatrix that multiplies each array entry by 2: mod = SourceModule(""" __global__ void doubleMatrix(float *a) { int idx = threadIdx.x + threadIdx.y*4; a[idx] *= 2; } """) The __global__ qualifier is a directive that indicates that the doubleMatrix function will be processed on the device. It will be just the compiler Cuda nvcc that will be used to perform this task. Let's look at the function's body, which is as follows: int idx = threadIdx.x + threadIdx.y*4; The idx parameter is the matrix index that is identified by the thread coordinates threadIdx.x and threadIdx.y. Then, the element matrix with the index idx is multiplied by 2: a[idx] *= 2; We also note that this kernel function will be executed once in 16 different threads. Both the variables threadIdx.x and threadIdx.y contain indices between 0 and 3 , and the pair[NV4]  is different for each thread. Threads scheduling is directly linked to the GPU architecture and its intrinsic parallelism. A block of threads is assigned to a single SM. Here, threads are further divided into groups called warps. The size of a warp depends on the architecture under consideration. The threads of the same warp are managed by the control unit called the warp scheduler. To take full advantage of the inherent parallelism of the SM, the threads of the same warp must execute the same instruction. If this condition does not occur, we speak of divergence of threads. If the same warp threads execute different instructions, the control unit cannot handle all the warps. It must follow the sequences of instructions for every single thread (or for homogeneous subsets of threads) in a serial mode. Let's observe how the thread block is divided in various warps—threads are divided by the value of threadIdx. The threadIdx structure consists of 3 fields: threadIdx.x, threadIdx.y, and threadIdx.z. Thread blocks subdivision: T(x,y), where x = threadIdx.x and y = threadIdx.y We can see again that the code in the kernel function will be automatically compiled by the nvcc cuda compiler. If there are no errors, a pointer to this compiled function will be created. In fact, the mod.get_function[NV5] ("doubleMatrix") returns an identifier to the function created func: func = mod.get_function("doubleMatrix ") To perform a function on the device, you must first configure the execution appropriately. This means that we need to determine the size of the coordinates to identify and distinguish the thread belonging to different blocks. This will be done using the block parameter inside the func call: func(a_gpu, block = (5, 5, 1)) The block = (5, 5, 1) tells us that we are calling a kernel function with a_gpu linearized input matrix and a single thread block of size, 5 threads in the x direction, 5 threads in the y direction, and 1 thread in the z direction, 16 threads in total. This structure is designed with parallel implementation of the algorithm of interest. The division of the workload results is an early form of parallelism that is sufficient and necessary to make use of the computing resources provided by the GPU. Once you've configured the kernel's invocation, you can invoke the kernel function that executes instructions in parallel on the device. Each thread executes the same code kernel. After the computation in the gpu device, we use an array to store the results: a_doubled = numpy.empty_like(a) cuda.memcpy_dtoh(a_doubled, a_gpu) Introducing PyOpenCL As for programming with PyCuda, the first step to build a program for PyOpenCL is the encoding of the host application. In fact, this is performed on the host computer (typically, the user's PC) and then, it dispatches the kernel application on the connected devices (GPU cards). The host application must contain five data structures, which are as follows: Device: This identifies the hardware where the kernel code must be executed. A PyOpenCL application can be executed not only on CPU and GPU cards but also on embedded devices such as FPGA (Field Programmable Gate Array). Program: This is a group of kernels. A program selects which kernel must be executed on the device. Kernel: This is the code to be executed on the device. A kernel is essentially a (C-like) function that enables it to be compiled for an execution on any device that supports OpenCL drivers. The kernel is the only way the host can call a function that will run on a device. When the host invokes a kernel, many work items start running on the device. Each work item runs the code of the kernel but works on a different part of the dataset. Command queue: Each device receives kernels through this data structure. A command queue orders the execution of kernels on the device. Context: This is a group of devices. A context allows devices to receive kernels and transfer data. PyOpenCL programming The preceding figure shows how these data structures can work in a host application. Let's remember again that a program can contain multiple functions that are to be executed on the device and each kernel encapsulates only a single function from the program. In this example, we show you the basic steps to build a PyOpenCL program. The task to be executed is the parallel sum of two vectors. In order to maintain a readable output, let's consider two vectors, each of one out of 100 elements. The resulting vector will be for each element's i[NV6] th, which is the sum of the ith element vector_a plus the ith element vector_b. Of course, to be able to appreciate the parallel execution of this code, you can also increase some orders of magnitude the size of the input vector_dimension:[NV7]  import numpy as np import pyopencl as cl import numpy.linalg as la vector_dimension = 100 vector_a = np.random.randint(vector_dimension, size=vector_dimension) vector_b = np.random.randint(vector_dimension, size=vector_dimension) platform = cl.get_platforms()[0] device = platform.get_devices()[0] context = cl.Context([device]) queue = cl.CommandQueue(context) mf = cl.mem_flags a_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_a) b_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_b) program = cl.Program(context, """ __kernel void vectorSum(__global const int *a_g, __global const int *b_g, __global int *res_g) { int gid = get_global_id(0); res_g[gid] = a_g[gid] + b_g[gid]; } """).build() res_g = cl.Buffer(context, mf.WRITE_ONLY, vector_a.nbytes) program.vectorSum(queue, vector_a.shape, None, a_g, b_g, res_g) res_np = np.empty_like(vector_a) cl.enqueue_copy(queue, res_np, res_g) print ("PyOPENCL SUM OF TWO VECTORS") print ("Platform Selected = %s" %platform.name ) print ("Device Selected = %s" %device.name) print ("VECTOR LENGTH = %s" %vector_dimension) print ("INPUT VECTOR A") print vector_a print ("INPUT VECTOR B") print vector_b print ("OUTPUT VECTOR RESULT A + B ") print res_np assert(la.norm(res_np - (vector_a + vector_b))) < 1e-5 The output from Command Prompt should be like this: C:Python CookBook Chapter 6 - GPU Programming with PythonChapter 6 - codes>python PyOpenCLParallellSum.py Platform Selected = NVIDIA CUDA Device Selected = GeForce GT 240 VECTOR LENGTH = 100 INPUT VECTOR A [ 0 29 88 46 68 93 81 3 58 44 95 20 81 69 85 25 89 39 47 29 47 48 20 86 59 99 3 26 68 62 16 13 63 28 77 57 59 45 52 89 16 6 18 95 30 66 19 29 31 18 42 34 70 21 28 0 42 96 23 86 64 88 20 26 96 45 28 53 75 53 39 83 85 99 49 93 23 39 1 89 39 87 62 29 51 66 5 66 48 53 66 8 51 3 29 96 67 38 22 88] INPUT VECTOR B [98 43 16 28 63 1 83 18 6 58 47 86 59 29 60 68 19 51 37 46 99 27 4 94 5 22 3 96 18 84 29 34 27 31 37 94 13 89 3 90 57 85 66 63 8 74 21 18 34 93 17 26 9 88 38 28 14 68 88 90 18 6 40 30 70 93 75 0 45 86 15 10 29 84 47 74 22 72 69 33 81 31 45 62 81 66 69 14 71 96 91 51 35 4 63 36 28 65 10 41] OUTPUT VECTOR RESULT A + B [ 98 72 104 74 131 94 164 21 64 102 142 106 140 98 145 93 108 90 84 75 146 75 24 180 64 121 6 122 86 146 45 47 90 59 114 151 72 134 55 179 73 91 84 158 38 140 40 47 65 111 59 60 79 109 66 28 56 164 111 176 82 94 60 56 166 138 103 53 120 139 54 93 114 183 96 167 45 111 70 122 120 118 107 91 132 132 74 80 119 149 157 59 86 7 92 132 95 103 32 129] In the first line of the code after the required module import, we defined the input vectors: vector_dimension = 100 vector_a = np.random.randint(vector_dimension, size= vector_dimension) vector_b = np.random.randint(vector_dimension, size= vector_dimension) Each vector contain 100 integers items that are randomly selected thought the NumPy function: np.random.randint(max integer , size of the vector) Then, we must select the device to run the kernel code. To do this, we must first select the platform using the get_platform() PyOpenCL statement: platform = cl.get_platforms()[0] This platform, as you can see from the output, corresponds to the NVIDIA CUDA platform. Then, we must select the device using the get_device() platform's method: device = platform.get_devices()[0] In the following steps, the context and the queue are defined, PyOpenCL provides the method context (device selected) and queue (context selected): context = cl.Context([device]) queue = cl.CommandQueue(context) To perform the computation in the device, the input vector must be transferred to the device's memory. So, two input buffers in the device memory must be created: mf = cl.mem_flags a_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_a) b_g = cl.Buffer(context, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=vector_b) Also, we prepare the buffer for the resulting vector: res_g = cl.Buffer(context, mf.WRITE_ONLY, vector_a.nbytes) Finally, the core of the script, the kernel code is defined inside a program as follows: program = cl.Program(context, """ __kernel void vectorSum(__global const int *a_g, __global const int *b_g, __global int *res_g) { int gid = get_global_id(0); res_g[gid] = a_g[gid] + b_g[gid]; } """).build() The kernel's name is vectorSum. The parameter list defines the data types of the input arguments (vectors of integers) and output data type (a vector of integer again). Inside the kernel, the sum of the two vectors is simply defined as: Initialize the vector index: int gid = get_global_id(0) Sum the vector's components: res_g[gid] = a_g[gid] + b_g[gid]; In OpenCL and PyOpenCL, buffers are attached to a context[NV8]  and are only moved to a device once the buffer is used on that device. Finally, we execute vectorSum in the device: program.vectorSum(queue, vector_a.shape, None, a_g, b_g, res_g) To visualize the results, an empty vector is built: res_np = np.empty_like(vector_a) Then, the result is copied into this vector: cl.enqueue_copy(queue, res_np, res_g) Finally, the results are displayed: print ("VECTOR LENGTH = %s" %vector_dimension) print ("INPUT VECTOR A") print vector_a print ("INPUT VECTOR B") print vector_b print ("OUTPUT VECTOR RESULT A + B ") print res_np To check the result, we use the assert statement. It tests the result and triggers an error if the condition is false: assert(la.norm(res_np - (vector_a + vector_b))) < 1e-5 Summary In this article we discussed about Asyncio, GPU programming with Python, PyCUDA, and PyOpenCL. Resources for Article: Further resources on this subject: Bizarre Python[article] Scientific Computing APIs for Python[article] Optimization in Python [article]

[box type="note" align="" class="" width=""]This article is a book excerpt from Learning Spark SQL written by Aurobindo Sarkar. This book gives an insight into the engineering practices used to design and build real-world Spark based applications. The hands on examples illustrated in the book will give you required confidence to work on future projects you encounter in Spark SQL. [/box] In the article, we shall talk about Spark SQL and its use in streaming applications. What are streaming applications? A streaming application is a program that has its necessary components downloaded as needed instead of being installed ahead of time on a computer. Application streaming is a method of delivering virtualized applications. Streaming applications are getting increasingly complex, because such computations don't run in isolation. They need to interact with batch data, support interactive analysis, support sophisticated machine learning applications, and so on. Typically, such applications store incoming event stream(s) on long-term storage, continuously monitor events, and run machine learning models on the stored data, while simultaneously enabling continuous learning on the incoming stream. They also have the capability to interactively query the stored data while providing exactly-once write guarantees, handling late arriving data, performing aggregations, and so on. These types of applications are a lot more than mere streaming applications and have, therefore, been termed as continuous applications. SparkSQL and Structured Streaming Before Spark 2.0, streaming applications were built on the concept of DStreams. There were several pain points associated with using DStreams. In DStreams, the timestamp was when the event actually came into the Spark system; the time embedded in the event was not taken into consideration. In addition, though the same engine can process both the batch and streaming computations, the APIs involved, though similar between RDDs (batch) and DStream (streaming), required the developer to make code changes. The DStream streaming model placed the burden on the developer to address various failure conditions, and it was hard to reason about data consistency issues. In Spark 2.0, Structured Streaming was introduced to deal with all of these pain points.  Structured Streaming is a fast, fault-tolerant, exactly-once stateful stream processing approach. It enables streaming analytics without having to reason about the underlying mechanics of streaming. In the new model, the input can be thought of as data from an append-only table (that grows continuously). A trigger specifies the time interval for checking the input for the arrival of new data. As shown in the following figure, the query represents the queries or the operations, such as map, filter, and reduce on the input, and result represents the final table that is updated in each trigger interval, as per the specified operation. The output defines the part of the result to be written to the data sink in each time interval.  The output modes can be complete, delta, or append, where the complete output mode means writing the full result table every time, the delta output mode writes the changed rows from the previous batch, and the append output mode writes the new rows only, Respectively: In Spark 2.0, in addition to the static bounded DataFrames, we have the concept of a continuous unbounded DataFrame. Both static and continuous DataFrames use the same API, thereby unifying streaming, interactive, and batch queries. For example, you can aggregate data in a stream and then serve it using JDBC. The high-level streaming API is built on the Spark SQL engine and is tightly integrated with SQL queries and the DataFrame/Dataset APIs. The primary benefit is that you use the same high-level Spark DataFrame and Dataset APIs, and the Spark engine figures out the incremental and continuous execution required for operations. Additionally, there are query management APIs that you can use to manage multiple, concurrently running, and streaming queries. For instance, you can list running queries, stop and restart queries, retrieve exceptions in case of failures, and so on. In the example code below, we use two bid files from the iPinYou Dataset as the source for our streaming data. First, we define our input records schema and create a streaming input DataFrame: scala> import org.apache.spark.sql.types._ scala> import org.apache.spark.sql.functions._ scala> import scala.concurrent.duration._ scala> import org.apache.spark.sql.streaming.ProcessingTime scala> import org.apache.spark.sql.streaming.OutputMode.Complete scala> val bidSchema = new StructType().add("bidid", StringType).add("timestamp", StringType).add("ipinyouid", StringType).add("useragent", StringType).add("IP", StringType).add("region", IntegerType).add("city", IntegerType).add("adexchange", StringType).add("domain", StringType).add("url:String", StringType).add("urlid: String", StringType).add("slotid: String", StringType).add("slotwidth", StringType).add("slotheight", StringType).add("slotvisibility", StringType).add("slotformat", StringType).add("slotprice", StringType).add("creative", StringType).add("bidprice", StringType) scala> val streamingInputDF = spark.readStream.format("csv").schema(bidSchema).option("header", false).option("inferSchema", true).option("sep", "t").option("maxFilesPerTrigger", 1).load("file:///Users/aurobindosarkar/Downloads/make-ipinyou-datamaster/ original-data/ipinyou.contest.dataset/bidfiles") Next, we define our query with a time interval of 20 seconds and the output mode as Complete: scala> val streamingCountsDF = streamingInputDF.groupBy($"city").count() scala> val query = streamingCountsDF.writeStream.format("console").trigger(ProcessingTime(20.s econds)).queryName("counts").outputMode(Complete).start() In the output, it is observed that the count of bids from each region gets updated in each time interval as new data arrives. The new bid files need to be dropped (or start with multiple bid files, as they will get picked up for processing one at a time based on the value of maxFilesPerTrigger) from the original Dataset into the bidfiles directory to see the updated results. Structured Streaming Internals Sources and incrementally executes the computation on it before writing it to the sink. In addition, any running aggregates required by your application are maintained as in-memory states backed by a Write-Ahead Log (WAL). The in-memory state data is generated and used across incremental executions. The fault tolerance requirements for such applications include the ability to recover and replay all data and metadata in the system. The planner writes offsets to a fault-tolerant WAL on persistent storage, such as HDFS, before execution as illustrated in the figure: In case the planner fails on the current incremental execution, the restarted planner reads from the WAL and re-executes the exact range of offsets required. Typically, sources such as Kafka are also fault-tolerant and generate the original transactions data, given the appropriate offsets recovered by the planner. The state data is usually maintained in a versioned, key-value map in Spark workers and is backed by a WAL on HDFS. The planner ensures that the correct version of the state is used to re-execute the transactions subsequent to a failure. Additionally, the sinks are idempotent by design, and can handle the re-executions without double commits of the output. Hence, an overall combination of offset tracking in WAL, state management, and fault-tolerant sources and sinks provide the end-to- end exactly-once guarantees. Summary SparkSQL provides one of the best platforms for implementing streaming applications. The internal architecture and the fault tolerant behavior implies that modern day developers who want to create data intensive applications with data streaming capabilities, will have to use the power of SparkSQL. If you liked our post, please be sure to check out Learning Spark SQL which consists of more useful techniques on data extraction and data analysis using Spark SQL.

In this article by Sinny Kumari, author of the book Linux Shell Scripting Essentials, we will cover I/O redirection pipes and filters. In day-to-day work, we come across different kinds of files such as text files, source code files from different programming languages (for example, file.sh, file.c, and file.cpp), and so on. While working, we more often perform various operations on files or directories such as searching for a given string or pattern, replacing strings, printing few lines of a file, and so on. Performing these operations is not easy if we have to do it manually. Manual searching for a string or pattern in a directory having thousands of files can take months, and has high chances of making errors. Shell provides many powerful commands to make our work easier, faster, and error-free. Shell commands have the ability to manipulate and filter text from different streams such as standard input, file, and so on. Some of these commands are grep, sed, head, tr, sort, and so on. Shell also comes with a feature of redirecting output from one command to another with the pipe ('|'). Using pipe helps to avoids creation of unnecessary temporary files. One of the best qualities of these commands is that they come along with the man pages. We can directly go to the man page and see what all features they provide by running the man command. Most of the commands have options such as --help to find the help usage and --version to know the version number of the command. This article will cover the following topics in detail: Standard I/O and error streams Redirecting the standard I/O and error streams Pipe and pipelines—connecting commands Regular expressions Filtering output using grep (For more resources related to this topic, see here.) Standard I/O and error streams In shell programming, there are different ways to provide an input (for example, via a keyboard and terminal) and display an output (for example, terminal and file) and error (for example, terminal), if any, during the execution of a command or program. The following examples show the input, output, and error while running the commands: The input from a user by a keyboard and the input obtained by a program via a standard input stream, that is terminal, is taken as follows: $ read -p "Enter your name:" Enter your name:Foo The output printed on the standard output stream, that is terminal, is as follows: $ echo "Linux Shell Scripting" Linux Shell Scripting The error message printed on the standard error stream, that is terminal, is as follows: $ cat hello.txt cat: hello.txt: No such file or directory When a program executes, by default, three files get opened with it which are stdin, stdout, and stderr. The following table provides a short description about them: File descriptor number File name Description 0 stdin This is standard input being read from the terminal 1 stdout This is standard output to the terminal 2 stderr This is standard error to the terminal File descriptors File descriptors are integer numbers representing opened files in an operating system. The unique file descriptor numbers are provided to each opened files. File descriptors' numbers go up from 0. Whenever a new process in Linux is created, then standard input, output, and error files are provided to it along with other needed opened files to process. To know what all open file descriptors are associated with a process, we will consider the following example: Run an application and get its process ID first. Consider running bash as an example to get PID of bash: $ pidof bash 2508 2480 2464 2431 1281 We see that multiple bash processes are running. Take one of the bash PID example, 2508, and run the following command: $ ls -l /proc/2508/fd total 0 lrwx------. 1 sinny sinny 64 May 20 00:03 0 -> /dev/pts/5 lrwx------. 1 sinny sinny 64 May 20 00:03 1 -> /dev/pts/5 lrwx------. 1 sinny sinny 64 May 19 23:22 2 -> /dev/pts/5 lrwx------. 1 sinny sinny 64 May 20 00:03 255 -> /dev/pts/5 We see that 0, 1, and 2 opened file descriptors are associated with process bash. Currently, all of them are pointing to /dev/pts/5. pts that is pseudo terminal slave. So, whatever we will do in this bash, input, output, and error related to this PID, output will be written to the /dev/pts/5 file. However, the pts files are pseudo files and contents are in memory, so you won't see anything when you open the file. Redirecting the standard I/O and error streams We have an option to redirect standard input, output, and errors, for example, to a file, another command, intended stream, and so on. Redirection is useful in different ways. For example, I have a bash script whose output and errors are displayed on a standard output—that is, terminal. We can avoid mixing an error and output by redirecting one of them or both to a file. Different operators are used for redirection. The following table shows some of operators used for redirection, along with its description: Operator Description >  This redirects a standard output to a file >>  This appends a standard output to a file <  This redirects a standard input from a file >& This redirects a standard output and error to a file >>& This appends a standard output and error to a file | This redirects an output to another command Redirecting standard output: An output of a program or command can be redirected to a file. Saving an output to a file can be useful when we have to look into the output in the future. A large number of output files for a program that runs with different inputs can be used in studying program output behavior. For example, showing redirecting echo output to output.txt is as follows: $ echo "I am redirecting output to a file" > output.txt $ We can see that no output is displayed on the terminal. This is because output was redirected to output.txt. The operator '>' (greater than) tells the shell to redirect the output to whatever filename mentioned after the operator. In our case, its output.txt: $ cat output.txt I am redirecting output to a file Now, let's add some more output to the output.txt file: $ echo "I am adding another line to file" > output.txt $ cat output.txt I am adding another line to file We noticed that the previous content of the output.txt file got erased and it only has the latest redirected content. To retain the previous content and append the latest redirected output to a file, use the operator '>>': $ echo "Adding one more line" >> output.txt $ cat output.txt I am adding another line to file Adding one more line We can also redirect an output of a program/command to another command in bash using the operator '|' (pipe): $ ls /usr/lib64/ | grep libc.so libc.so libc.so.6 In this example, we gave the output of ls to the grep command using the '|' (pipe) operator, and grep gave the matching search result of the libc.so library: Redirecting standard input Instead of getting an input from a standard input to a command, it can be redirected from a file using the < (less than) operator. For example, we want to count the number of words in the output.txt file created from the Redirecting standard output section: $ cat output.txt I am adding another line to file Adding one more line $ wc -w < output.txt 11 We can sort the content of output.txt: $ sort < output.txt # Sorting output.txt on stdout Adding one more line I am adding another line to file We can also give a patch file as an input to the patch command in order to apply a patch.diff in a source code. The command patch is used to apply additional changes made in a file. Additional changes are provided as a diff file. A diff file contains the changes between the original and the modified file by running the diff command. For example, I have a patch file to apply on output.txt: $ cat patch.diff # Content of patch.diff file 2a3 > Testing patch command $ patch output.txt < patch.diff # Applying patch.diff to output.txt $ cat output.txt # Checking output.txt content after applying patch I am adding another line to file Adding one more line Testing patch command Redirecting standard error There is a possibility of getting an error while executing a command/program in bash because of different reasons such as invalid input, insufficient arguments, file not found, bug in program, and so on: $ cd /root # Doing cd to root directory from a normal user bash: cd: /root/: Permission denied Bash prints the error on a terminal saying, permission denied. In general, errors are printed on a terminal so that it's easy for us to know the reason of an error. Printing both the errors and output on the terminal can be annoying because we have to manually look into each line and check whether the program encountered any error: $ cd / ; ls; cat hello.txt; cd /bin/; ls *.{py,sh} We ran a series of commands in the preceding section. First cd to /, ls content of /, cat file hello.txt, cd to /bin and see files matching *.py and *.sh in /bin/. The output will be as follows: bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var cat: hello.txt: No such file or directory alsa-info.sh kmail_clamav.sh sb_bnfilter.py sb_mailsort.py setup-nsssysinit.sh amuFormat.sh kmail_fprot.sh sb_bnserver.pysb_mboxtrain.py struct2osd.sh core_server.py kmail_sav.shsb_chkopts.py sb_notesfilter.py We see that hello.txt doesn't exist in the / directory and because of this there is an error printed on the terminal as well, along with other output. We can redirect the error as follows: $ (cd / ; ls; cat hello.txt; cd /bin/; ls *.{py,sh}) 2> error.txt bin boot dev etc home lib lib64 lost+found media mnt opt proc rootrun sbin srv sys tmp usr var alsa-info.sh kmail_clamav.sh sb_bnfilter.py sb_mailsort.py setup-nsssysinit.sh amuFormat.sh kmail_fprot.sh sb_bnserver.pysb_mboxtrain.py struct2osd.sh core_server.py kmail_sav.shsb_chkopts.py sb_notesfilter.py We can see that the error has been redirected to the error.txt file. To verify, check the error.txt content: $ cat error.txt cat: hello.txt: No such file or directory Multiple redirection: We can redirect stdin, stdout, and stderr together in a command or script or a combination of some of them. The following command redirects both stdout and stder: $ (ls /home/ ;cat hello.txt;) > log.txt 2>&1 Here, stdout is redirected to log.txt and error messages are redirected to log.txt as well. In 2>&1, 2> means redirect an error and &1 means redirect to stdout. In our case, we have already redirected stdout to the log.txt file. So, now both the stdout and stderr outputs will be written into log.txt and nothing will be printed on the terminal. To verify, we will check the content of log.txt: $ cat log.txt lost+found sinny cat: hello.txt: No such file or directory The following example shows the stdin, stdout, and stderr redirection: $ cat < ~/.bashrc > out.txt 2> err.txt Here, the .bashrc file present in the home directory acts as an input to the cat command and its output is redirected to the out.txt file. Any error encountered in between is redirected to the err.txt file. The following bash script will explain stdin, stdout, stderr, and their redirection with even more clarity: #!/bin/bash # Filename: redirection.sh # Description: Illustrating standard input, output, error # and redirecting them ps -A -o pid -o command > p_snapshot1.txt echo -n "Running process count at snapshot1: " wc -l < p_snapshot1.txt echo -n "Create a new process with pid = " tail -f /dev/null & echo $! # Creating a new process echo -n "Running process count at snapshot2: " ps -A -o pid -o command > p_snapshot2.txt wc -l < p_snapshot2.txt echo echo "Diff bewteen two snapshot:" diff p_snapshot1.txt p_snapshot2.txt This script saves two snapshots of running all the currently running processes in the system and generates diff. The output after running the process will look somewhat as follows: $ sh redirection.sh Running process count at snapshot1: 246 Create a new process with pid = 23874 Running process count at snapshot2: 247 Diff bewteen two snapshot: 246c246,247 < 23872 ps -A -o pid -o command --- > 23874 tail -f /dev/null > 23875 ps -A -o pid -o command Pipe and pipelines – connecting commands The outputs of the programs are generally saved in files for further use. Sometimes, temporary files are created in order to use an output of a program as an input to another program. We can avoid creating temporary files and feed the output of a program as an input to another program using bash pipe and pipelines. Pipe The pipe denoted by the operator | connects the standard output of a process in the left to the standard input in the right process by inter process communication mechanism. In other words, the | (pipe) connects commands by providing the output of a command as the input to another command. Consider the following example: $ cat /proc/cpuinfo | less Here, the cat command, instead of displaying the content of the /proc/cpuinfo file on stdout, passes its output as an input to the less command. The less command takes the input from cat and displays on the stdout per page. Another example using pipe is as follows: $ ps -aux | wc -l # Showing number of currently running processes in system 254 Pipeline Pipeline is a sequence of programs/commands separated by the operator ' | ' where the output of execution of each command is given as an input to the next command. Each command in a pipeline is executed in a new subshell. The syntax will be as follows: command1 | command2 | command3 … Examples showing pipeline are as follows: $ ls /usr/lib64/*.so | grep libc | wc -l 13 Here, we are first getting a list of files from the /usr/lib64 directory that has the.so extension. The output obtained is passed as an input to the next grep command to look for the libc string. The output is further given to the wc command to count the number of lines. Regular expression Regular expression (also known as regex or regexp) provides a way of specifying a pattern to be matched in a given big chunk of text data. It supports a set of characters to specify the pattern. It is widely used for a text search and string manipulation. A lot of shell commands provide an option to specify regex such as grep, sed, find, and so on. The regular expression concept is also used in other programming languages such as C++, Python, Java, Perl, and so on. Libraries are available in different languages to support regular expression's features. Regular expression metacharacters The metacharacters used in regular expressions are explained in the following table: Metacharacters Description * (Asterisk) This matches zero or more occurrences of the previous character + (Plus) This matches one or more occurrences of the previous character ? This matches zero or one occurrence of the previous element . (Dot) This matches any one character ^ This matches the start of the line $ This matches the end of line [... ] This matches any one character within a square bracket [^... ] This matches any one character that is not within a square bracket | (Bar) This matches either the left side or the right side element of | {X} This matches exactly X occurrences of the previous element {X,} This matches X or more occurrences of the previous element {X,Y} This matches X to Y occurrences of the previous element (...) This groups all the elements < This matches the empty string at the beginning of a word > This matches the empty string at the end of a word This disables the special meaning of the next character Character ranges and classes When we look into a human readable file or data, its major content contains alphabets (a to z) and numbers (0-9). While writing regex for matching a pattern consisting of alphabets or numbers, we can make use character ranges or classes. Character ranges We can use character ranges in a regular expression as well. We can specify a range by a pair of characters separated by a hyphen. Any characters that fall in between that range, inclusive, are matched. Character ranges are enclosed inside square brackets. The following table shows some of character ranges: Character range Description [a-z] This matches any single lowercase letter from a to z [A-Z] This matches any single uppercase letter from A to Z [0-9] This matches any single digit from 0 to 9 [a-zA-Z0-9] This matches any single alphabetic or numeric characters [h-k] This matches any single letter from h to k [2-46-8j-lB-M] This matches any single digit from 2 to 4 or 6 to 8 or any letter from j to l or B to M Character classes: Another way of specifying a range of character matches is by using Character classes. It is specified within the square brackets [:class:]. The possible class value is mentioned in the following table: Character Class Description [:alnum:] This matches any single alphabetic or numeric character; for example, [a-zA-Z0-9] [:alpha:] This matches any single alphabetic character; for example, [a-zA-Z] [:digit:] This matches any single digit; for example, [0-9] [:lower:] This matches any single lowercase alphabet; for example, [a-z] [:upper:] This matches any single uppercase alphabet; for example, [A-Z] [:blank:] This matches a space or tab [:graph:] This matches a character in the range of ASCII—for example 33-126—excluding a space character [:print:] This matches a character in the range of ASCII—for example. 32-126—including a space character [:punct:] This matches any punctuation marks such as '?', '!', '.', ',', and so on [:xdigit:] This matches any hexadecimal characters; for example, [a-fA-F0-9] [:cntrl:] This matches any control characters Creating your own regex: In the previous sections of regular expression, we discussed about metacharacters, character ranges, character class, and their usage. Using these concepts, we can create powerful regex that can be used to filter out text data as per our need. Now, we will create a few regex using the concepts we have learned. Matching dates in mm-dd-yyyy format We will consider our valid date starting from UNIX Epoch—that is, 1st January 1970. In this example, we will consider all the dates between UNIX Epoch and 30th December 2099 as valid dates. An explanation of forming its regex is given in the following subsections: Matching a valid month 0[1-9] matches 01st to 09th month 1[0-2] matches 10th, 11th, and 12th month '|' matches either left or right expression Putting it all together, the regex for matching a valid month of date will be 0[1-9]|1[0-2]. Matching a valid day 0[1-9] matches 01st to 09th day [12][0-9] matches 10th to 29th day 3[0-1] matches 30th to 31st day '|' matches either left or right expression 0[1-9]|[12][0-9]|3[0-1] matches all the valid days in a date Matching the valid year in a date 19[7-9][[0-9] matches years from 1970 to 1999 20[0-9]{2} matches years from 2000 to 2099 '|' matches either left or right expression 19[7-9][0-9]|20[0-9]{2} matches all the valid years between 1970 to 2099 Combining valid months, days, and years regex to form valid dates Our date will be in mm-dd-yyyy format. By putting together regex formed in the preceding sections for months, days, and years, we will get regex for the valid date: (0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[0-1])-(19[7-9][0-9]|20[0-9]{2}) There is a nice website http://regexr.com/, where you can also validate regular expression. The following screenshot shows the matching of the valid date among the given input: Regex for a valid shell variable A valid variable name can contain a character from alphanumeric and underscore, and the first letter of the variable can't be a digit. Keeping these rules in mind, a valid shell variable regex can be written as follows: ^[_a-zA-Z][_a-zA-Z0-9]*$ Here, ^ (caret) matches the start of a line. The regex [_a-zA-Z] matches _ or any upper or lower case alphabet [_a-zA-Z0-9]* matches zero or multiple occurrences of _,any digit or upper and lower case alphabet $ (Dollar) matches the end of the line. In character class format, we can write regex as ^[_[:alpha:]][_[:alnum:]]*$. The following screenshot shows valid shell variables using regex formed: Enclose regular expression in single quotes (') to avoid pre-shell expansion. Use back slash () before a character to escape the special meaning of metacharacters. Metacharacters such as ?, +, {, |, (, and ) are known to be extended regex. They lose their special meaning when used in basic regex. To avoid this, use them with backslash '?', '+', '{', '|', '(', and ')'. Filtering an output using grep One of the powerful and widely used command in shell is grep. It searches in an input file and matches lines in which the given pattern is found. By default, all the matched patterns are printed on stdout that is usually terminal. We can also redirect the matched output to other streams such as file. Instead of giving an input from a file, grep can also take the input from the redirected output of the command executed on the left-hand side of '|'. Syntax The syntax of using the grep command is as follows: grep [OPTIONS] PATTERN [FILE...] Here, FILE can be multiple files for a search. If no file is given as an input for a search, it will search the standard input. PATTERN can be any valid regular expression. Put PATTERN within single quotes (') or double quotes (") as per need. For example, use single quotes (') to avoid any bash expansion and double quotes (") for expansion. A lot of OPTIONS are available in grep. Some of the important and widely used options are discussed in the following table.   Option Usage -i This enforces case insensitive match in both pattern and input file(s) -v This displays the non-matching line -o This displays only the matched part in the matching line -f FILE This obtains a pattern from a file, one per line -e PATTERN This specifies multiple search pattern -E This considers pattern as an extended regex (egrp) -r This reads all the files in a directory recursively, excluding resolving of symbolic links unless explicitly specified as an input file -R This reads all the files in a directory recursively and resolving symbolic if any -a This processes binary file as a text file -n This prefixes each matched line along with a line number -q Don't print anything on stdout -s Don't print error messages -c This prints the count of matching lines of each input file -A NUM This prints NUM lines after the actual string match. No effect with the -o option -B NUM This prints NUM lines before the actual string match. No effect with the -o option -C NUM This prints NUM lines after and before the actual string match. No effect with the -o option  Looking for a pattern in a file: A lot of times we have to search for a given string or a pattern in a file. The grep command provides us the capability to do it in a single line. Let's see the following example: The input file for our example will be input1.txt: $ cat input1.txt # Input file for our example This file is a text file to show demonstration of grep command. grep is a very important and powerful command in shell. This file has been used in chapter 2 We will try to get the following information from the input1.txt file using the grep command: Number of lines Line starting with a capital letter Line ending with a period (.) Number of sentences Searching sub-string "sent" Lines that don't have a periodNumber of times the string "file" is used  The following shell script demonstrates how to do the above mentioned tasks: #!/bin/bash #Filename: pattern_search.sh #Description: Searching for a pattern using input1.txt file echo "Number of lines = `grep -c '.*' input1.txt`" echo "Line starting with capital letter:" grep -c ^[A-Z].* input1.txt echo echo "Line ending with full stop (.):" grep '.*.$' input1.txt echo echo -n "Number of sentence = " grep -c '.' input1.txt echo "Strings matching sub-string sent:" grep -o "sent" input1.txt echo echo "Lines not having full stop are:" grep -v '.' input1.txt echo echo -n "Number of times string file used: = " grep -o "file" input1.txt | wc -w  The output after running the pattern_search.sh shell script will be as follows: Number of lines = 4 Line starting with capital letter: 2 Line ending with full stop (.): powerful command in shell. Number of sentence = 2 Strings matching sub-string sent: Lines not having full stop are: This file is a text file to show demonstration This file has been used in chapter 2 Number of times string file used: = 3 Looking for a pattern in multiple files The grep command also allow us to search for a pattern in multiple files as an input. To explain this in detail, we will head directly to the following example. The input files, in our case, will be input1.txt and input2.txt. We will reuse the content of the input1.txt file from the previous example. The content of input2.txt is as follows: $ cat input2.txt Another file for demonstrating grep CommaNd usage. It allows us to do CASE Insensitive string test as well. We can also do recursive SEARCH in a directory using -R and -r Options. grep allows to give a regular expression to search for a PATTERN. Some special characters like . * ( ) { } $ ^ ? are used to form regexp. Range of digit can be given to regexp e.g. [3-6], [7-9], [0-9] We will try to get the following information from the input1.txt and input2.txt files using the grep command: Search for the string command Case-insensitive search of the string command Print the line number where the string grep matches Search for punctuation marks Print one line followed by the matching lines while searching for the string important  The following shell script demonstrates how to follow the preceding steps: #!/bin/bash # Filename: multiple_file_search.sh # Description: Demonstrating search in multiple input files echo "This program searches in files input1.txt and input2.txt" echo "Search result for string "command":" grep "command" input1.txt input2.txt echo echo "Case insensitive search of string "command":" # input{1,2}.txt will be expanded by bash to input1.txt input2.txt grep -i "command" input{1,2}.txt echo echo "Search for string "grep" and print matching line too:" grep -n "grep" input{1,2}.txt echo echo "Punctuation marks in files:" grep -n [[:punct:]] input{1,2}.txt echo echo "Next line content whose previous line has string "important":" grep -A 1 'important' input1.txt input2.txt The following screenshot is the output after running the shell script pattern_search.sh. The matched pattern string has been highlighted: A few more grep usages The following subsections will cover a few more usages of the grep command. Searching in a binary file So far, we have seen all the grep examples running on text files. We can also search for a pattern in binary files using grep. For this, we have to tell the grep command to treat a binary file as a text file too. The option -a or –text tells grep to consider a binary file as a test file. We know that the grep command itself is a binary file that executes and gives a search result. One of options in grep is --text. The string --text should be somewhere available in the grep binary file. Let's search for it as follows: $ grep --text '--text' /usr/bin/grep -a, --text equivalent to –binary-files=text We saw that the string --text is found in the search path /usr/bin/grep. The character backslash ('') is used to escape its special meaning. Now, let's search for the -w string in the wc binary. We know that the wc command has an option -w that counts the number of words in an input text. $ grep -a '-w' /usr/bin/wc -w, --words print the word counts Searching in a directory We can also tell grep to search into all files/directories in a directory recursively using the option -R. This avoids the hassle of specifying each file as an input text file to grep. For example, we are interested in knowing at how many places #include <stdio.h> is used in a standard include directory: $ grep -R '#include <stdio.h>' /usr/include/ | wc -l 77 This means that the #include <stdio.h> string is found at 77 places in the /usr/include directory. In another example, we want to know how many python files (the extension .py) in /usr/lib64/python2.7/ does "import os". We can check that as follows: $ grep -R "import os" /usr/lib64/python2.7/*.py | wc -l 93 Excluding files/directories from search We can also specify the grep command to exclude a particular directory or file from search. This is useful when we don't want grep to look into a file or directory that has some confidential information. This is also useful in the case where we are sure that searching into a certain directory will be of no use. So, excluding them will reduce search time. Suppose, there is a source code directory called s0, which uses the git version control. Now, we are interested in searching for a text or pattern in source files. In this case, searching in the .git subdirectory will be of no use. We can exclude .git from search as follows: $ grep -R --exclude-dir=.git "search_string" s0 Here, we are searching for the search_string string in the s0 directory and telling grep to not to search in the .git directory. Instead of excluding a directory, to exclude a file, use the --exclude-from=FILE option. Display a filename with a matching pattern In some use-cases, we don't bother with where the search matched and at how many places the search matched in a file. Instead, we are interested in knowing only the filename where at least one search matched. For example, I want to save filenames that have a particular search pattern found in a file, or redirect to some other command for further processing. We can achieve this using the -l option: $ grep -Rl "import os" /usr/lib64/python2.7/*.py > search_result.txt $ wc -l search_result.txt 79 This example gets name of the file in which import os is written and saves result in file search_result.txt. Matching an exact word The exact matching of the word is also possible using word boundary that is b on both the sides of the search pattern. Here, we will reuse the input1.txt file and its content: $ grep -i --color "bab" input1.txt The --color option allows colored printing of the matched search result. The "bab" option tells grep to only look for the character a that is alone. In search results, it won't match the character a present as a sub-string in a string. The following screenshot shows the output: This is an input file. It conatins special character like ?, ! etc &^var is an invalid shll variable. _var1_ is a valid shell variable To delete characters other than alphanumeric, newline, and white-space, we can run the following command: tr -cd '[:alnum:] n' < tr2.txt This is an input file It conatins special character like etc var is an invalid shll variable var1 is a valid shell variable Summary After reading this article, you know how to provide an input to commands and print or save its result. You are also familiar with redirecting an output and input from one command to another. Now, you can easily search, replace strings or pattern in a file, and filter out data based on needs. From this article, we now have a good control on transforming/filtering text data. Resources for Article: Further resources on this subject: Linux Shell Scripting [article] Embedded Linux and Its Elements [article] Getting started with Cocos2d-x [article]

This article is the second part of a series of articles, please refer to Part 1 for learning how to create a simple portfolio website using ChatGPT in a few stepsIn my last blog, we created a basic personal portfolio app for Olivia with ChatGPT. We’ll use this as the starting point for our next adventure. We’re going to be adding a simple chatbot to our website and configuring it to deal with certain questions. Of course, we could do this ourselves, but let’s see how we can utilize ChatGPT to accomplish this task faster. We’re going to do so with just four prompts! Let’s start with the end result, so you know what we’re working towards:Image 1: End result of the project In the right bottom corner, you see a pop-up that says Ask me anything. Well, you can’t ask it anything, but it can answer the following common questions:Where can I contact you?Can I contact you through Twitter?Can I contact you through LinkedIn?Where can I see your projects? Of course, we can elaborate and do even more with it. We actually wanted to use a free API for this, and we were stuck with CDN options due to the nature of our project. Luckily, we have a digital friend who has some options for us. Let’s see how this started.Deciding which service to use (with ChatGPT)It’s best to be quite descriptive when prompting. This is the first prompt that was used to start implementing the chatbot:I have an existing portfolio site, using HTML/CSS/JS. I'm not using any package managers, so any requirements will have to use a CDN. I want to add a chatbot to my site using a Natural Language AI. If it is possible I'd want the API I'll use to be free. Can you give me some options as to how I can achieve this? ChatGPT had some ideas: Certainly! There are several options available for adding a chatbot with Natural Language AI to your existing portfolio site using HTML/CSS/JS and integrating it through a CDN. Here are a few options that offer free plans: 1. IBM Watson Assistant: IBM Watson Assistant is a powerful NLP service that allows you to build chat bots. You can integrate it into your website using the Watson Assistant API. IBM offers a free tier that provides a limited number of API requests per month. You can find more information and documentation on the IBM Watson Assistant website. 2. Dialogflow (formerly API.AI): Dialogflow, now part of Google Cloud, offers a free tier that allows you to create conversational agents and integrate them into your website. Dialogflow provides a JavaScript library that you can use to communicate with the API. You can find documentation and examples on the Dialogflow website. 3. Wit.ai: Wit.ai, which is also owned by Facebook, provides a free NLP service for building chatbots. You can integrate it into your website using the Wit.ai API. Wit.ai offers a JavaScript SDK that you can use to interact with the API. You can find more information on the Wit.ai website. 4. Microsoft Azure Bot Service: Microsoft Azure Bot Service offers a free tier that allows you to create and deploy chatbots. You can use the Bot Service REST API to communicate with your bot from your website. Microsoft provides documentation and examples on its Azure Bot Service website. Remember to review the terms and limitations of the free plans for each service to ensure they meet your requirements. Additionally, be aware that some services may have usage limitations or require you to upgrade to a paid plan for higher usage levels or advanced features. Choose the option that best suits your needs, and refer to the respective documentation for implementation details specific to each service. After this, it was research time. This actually took a little while. IBM wasn’t properly explored, because the sign-up failed due to the Dutch valid phone number not passing validation. Azure seemed great, but a bit too advanced for what we needed right now. The chatbot service chosen was Wit.ai. On there we can sign up, get an access token and specify how to deal with certain inputs. After figuring that out and making a little code start ourselves. It was time to start adjusting the code with ChatGPT. Prompting for codeEnter the following prompt, to begin with: Alright, I set up a wit.ai project, and it seems to work. Now I have the following snippet:   <section id="chat-bot">    <h2>Ask me anything!</h2>    <input id="chat-prompt" type="text">  </section> I'd like to create a chatbot with this. I can write the responses myself, but can you for now create some JavaScript code that sends an HTTP request to the wit.ai server and parses the result? For now, just print it to the console. It gave us a great place to start and some additional implementation guidelines. We’ve only included the code snippet here:// Function to send the user's message to Wit.ai and parse the response async function sendMessage(message) { // Replace 'YOUR_WITAI_ACCESS_TOKEN' with your actual Wit.ai access token const accessToken = 'YOUR_WITAI_ACCESS_TOKEN'; const apiUrl = `https://api.wit.ai/message?v=20210525&q=${encodeURIComponent(message)}`; try {    const response = await fetch(apiUrl, {      headers: {        Authorization: `Bearer ${accessToken}`      }    });    if (response.ok) {      const data = await response.json();      console.log('Wit.ai Response:', data);      // Process the data and handle the response as needed      // For example, extract intents, entities, or generate appropriate responses    } else {      console.log('Request to Wit.ai failed with status:', response.status);    } } catch (error) {    console.log('An error occurred while communicating with Wit.ai:', error); } } // Event listener for handling user input const inputElement = document.getElementById('chat-prompt'); inputElement.addEventListener('keydown', async (event) => { if (event.key === 'Enter') {    const message = event.target.value.trim();    if (message) {      console.log('User Input:', message);      await sendMessage(message);      event.target.value = ''; // Clear the input field    } } }); It also instructed us to take the following steps:Replace YOUR_WITAI_ACCESS_TOKEN with our actual access tokenPress enter after typing your question and it then logs the response to the console Make sure to put the JavaScript code in the <script> tag This is what it looked like:Image 2: practical output after the first few prompts This was already quite good, but of course, we don’t want the response in the console, but displayed in the chat window. And the same goes for our previous questions, it should become a chat. Let’s see if we can move toward that.Making it into an actual conversationIt’s always a good idea to be specific in your prompt. You can see the following prompt does quite a bit of the heavy lifting and explains clearly what type of work is needed from ChatGPT. This helps to achieve the result: I've modified the HTML a little bit:   <section id="chat-bot">    <ol id="chat-history"></ol>    <h2>Ask me anything!</h2>    <input id="chat-prompt" type="text">  </section> Can you write some logic which can populate the `chat-history` element? I need 3 things: - A history as a JavaScript object- A function which can generate the content from the history- A function which can add the question, and the response, to the history in JavaScript For now just print the top intent and all the entities. This did quite a bit of the work already. You can see the result below, the history and the answers of the two questions asked appear on top of the input box.Image 3: Answers and history appearing on top of the Input box                        It’s no secret that I’m not a UX or design expert, but even I can see that this is not ideal. So let’s see if we can improve the design.Asking ChatGPT to Style the ChatbotLet’s see if we can improve the design. Normally chatbots are not on the top left, but the bottom right. Also, it would be nice if the history could go inside the box. Of course, we want to use the same orange as for the rest of our portfolio. Here’s the prompt: Can you generate some CSS to style the element? I'd like the following: - The chat should be at the bottom right of the page.- The chat should not impact the rest of the page.- Each question and response should have a chat bubble.- The theme color is #f37041.This gave us the CSS and some context as output. Here’s the CSS:#chat-bot { position: fixed; bottom: 20px; right: 20px; z-index: 9999; background-color: #f37041; color: #fff; font-family: Arial, sans-serif; border-radius: 5px; padding: 10px; box-shadow: 0 2px 5px rgba(0, 0, 0, 0.3); } #chat-history { list-style: none; margin: 0; padding: 0; max-height: 200px; overflow-y: auto; } .question, .response { display: inline-block; max-width: 80%; padding: 8px; border-radius: 10px; margin-bottom: 5px; } .question { background-color: #fff; color: #333; text-align: right; } .response { background-color: #f37041; color: #fff; text-align: left; } #chat-prompt { width: 100%; padding: 8px; margin-top: 10px; border: none; border-radius: 5px; } #chat-prompt:focus { outline: none; } #chat-prompt::placeholder { color: #ccc; } And this did quite the trick. Here’s what it looks like now: Image4: The final output If you’d like to see all the responses and the versions of the personal portfolio after each step, you can check out this GitHub repo: https://github.com/BrightBoost/portfolio-with-ai-prompt-chatgptSummaryIn conclusion, integrating a basic chatbot into your personal portfolio website can significantly enhance user engagement and provide a unique interactive experience. By leveraging GPT prompts, you can effortlessly develop a conversational AI agent that understands and responds to user queries. Adding this functionality not only showcases your technical skills but also demonstrates your ability to create dynamic and user-centric web experiences. So go ahead, implement a chatbot and elevate your portfolio website to the next level. Happy prompting!Author Bio Maaike van Putten is an experienced software developer and Pluralsight, LinkedIn Learning, Udemy, and Bright Boost instructor. She has a passion for software development and helping others get to the next level in their career.LinkedInTraining courses   

There are many ways you can customize your Slack program, but with a little bit of programming you can create things that can provide you with all sorts of productivity-enhancing options. In this post, you are going to learn how to use Python to create a custom progress bar that will run in your very own Slack channel. You can use this progress bar in so many different ways. We won’t get into the specifics beyond the progress bar, but one idea would be to use it to broadcast the progress of a to-do list in Trello. That is something that is easy to set up in Slack. Let’s get started. Real-time notifications are a great way to follow along with the events of the day. For ever-changing and long-running events though, a stream of update messages quickly becomes spammy and overwhelming. If you make a typo in a sent message, or want to append additional details, you know that you can go back and edit it without sending a new chat(and subsequently a new notification for those in the channel); so why can't your bot friends do the same? As you may have guessed by now, they can! This program will make use of Slack’s chat.update method to create a dynamic progress bar with just one single-line message. Save the below script to a file namedprogress.py, updating the SLACK_TOKEN and SLACK_CHANNEL variables at the top with your configured token and desired channel name. from time import sleep from slacker import Slacker from random importrandint SLACK_TOKEN = '<token>' SLACK_CHANNEL = '#general' classSlackSimpleProgress(object): def__init__(self, slack_token, slack_channel): self.slack = Slacker(slack_token) self.channel = slack_channel res = self.slack.chat.post_message(self.channel, self.make_bar(0)) self.msg_ts = res.body['ts'] self.channel_id = res.body['channel'] def update(self, percent_done): self.slack.chat.update(self.channel_id, self.msg_ts, self.make_bar(percent_done)) @staticmethod defmake_bar(percent_done): return'%s%s%%' % ((round(percent_done / 5) * chr(9608)), percent_done) if__name__ == '__main__': progress_bar = SlackSimpleProgress(SLACK_TOKEN, SLACK_CHANNEL) # initialize the progress bar percent_complete = 1 whilepercent_complete<100: progress.update(percent_complete) percent_complete += randint(1,10) sleep(1) progress_bar.update(100) Run the script with a simple python progress.py and, like magic, your progress bar will appear in the channel, updating at regular intervals until completion:   How It Works In the last six lines of the script: A progress bar is initially created with SlackSimpleProgress. The update method is used to refresh the current position of the bar once a second. percent_complete is slowly incremented by a random amount using Python’s random.randint. We set the progress bar to 100% when completed. Extending With the basic usage down, the same script can be updated to serve a real purpose. Say we're copying a large amount of files between directories for a backup—we'll replace the __main__ part of the script with: importos import sys importshutil src_path = sys.argv[0] dest_path = sys.argv[1] all_files = next(os.walk('/usr/lib'))[2] progress_bar = SlackSimpleProgress(SLACK_TOKEN, SLACK_CHANNEL) # initialize the progress bar files_copied = 0 forfile in all_files: src_file = '%s/%s' % (src_path, file) dest_file = '%s/%s' % (dest_path, file) print('copying %s to %s' % (src_file, dest_file)) # print the file being copied shutil.copy2(src_file, dest_file) files_copied += 1# increment files copied percent_done = files_copied / len(all_files) * 100# calculate percent done progress_bar.update(percent_done) The script can now be run with two arguments: python progress.py /path/to/files /path/to/destination. You'll see the name of each file copied in the output of the script, and your teammates will be able to follow along with the status of the copy as it progresses in your Slack channel! I hope you find many uses for ways to implement this progress bar in Slack! About the author Bradley Cicenas is a New York City-based infrastructure engineer with an affinity for microservices, systems design, data science, and stoops.

  (For more resources on iPhone JavaScript, see here.) Introduction The mashup applications allow us to exchange data with other web applications or services. Web 2.0 applications provide this feature through different mechanisms. Currently, some of the most popular websites like YouTube, Flickr, and Twitter provide a way for exchanging data through their API's. From the point of view of the user interfaces, mashups allow us to build rich interfaces for our application. The first recipe for this article will cover embedding a standard RSS feed information. Later in our application we'll delve into YouTube, Facebook, Twitter, and Flickr and build some interesting mashup web applications for the iPhone. Embedding an RSS feed Our goal for this recipe will be to read an RSS feed and present the information provided n our application. In practice, we're going to use a feed offered by The New York Times newspaper, where each item provides a summary and a link to the original web page where the information resides. You could also choose another RSS feed for testing this recipe. The code for this recipe can be found at code/ch10/rss.html in the code bundle provided on the Packtpub site. Getting ready Make sure iWebKit is installed in your computer before continuing. How to do it... As you've learned in the previous recipes, you need to create an XHTML file with the required headers for loading the files provided by iWebKit: <link href="../iwebkit/css/style.css" rel="stylesheet" media="screen" type="text/css" /><scriptsrc="../iwebkit/javascript/functions.js" type="text/javascript"></script> The second step will be to build our simple user interface containing only a top bar and an unordered list with one item. The top bar is added with the following lines: <div id="topbar"> <div id="title">RSS feed</div></div> To add the unordered list , use the following code: <div id="content"> <ul class="pageitem"> <li class="textbox"> <p> <script src="http://rssxpress.ukoln.ac.uk/lite/ viewer/?rss=http://www.nytimes.com/services/xml/ rss/nyt/HomePage.xml" type="text/javascript"></script> </p> </li> </ul></div> Finally, you should add the code for closing the body and html tags and save the new file as rss.html. After loading your new application, you will see a screen, as shown in the screenshot: If you click on one of the items, Safari Mobile will open the web page for the article, as shown in the following screenshot: How it works... For avoiding complexity and keeping our recipe as simple as possible, we used a free web service provided by RSSxpress. This service is called RSSxpressLite and it works by returning a chunk of JavaScript code. This code inserts an HTML table, containing a summary and a link for each item provided by the referenced RSS feed. Thanks to this web service, we don't need to parse the response of the original feed; RSSxpressLite does the job for us. If the mentioned web service returns the code that we need, you should only write a small line of JavaScript code referring to the web service through its URL and pass as a parameter the RSS feed for displaying information. There's more... For learning more about RSSxpressLite, take a look at http://rssxpress.ukoln.ac.uk/lite/include/. Opening a YouTube video It is safe to say that everyone who uses the Internet knows of YouTube. It is one of the most popular websites in the world. Millions of people use YouTube to watch videos through an assortment of devices, such as PC's, tablets, and smartphones. Apple's devices are not an exception and of course we can watch YouTube videos on the iPhone and iPad. In this case, we're going to load a YouTube video when the user clicks on a specific button. The ink will open a new web page, which allows us to play it. The simple XHTML recipe can be found at code/ch10/youtube.html in the code bundle provided on the Packtpub site. Getting ready This recipe only requires the UiUIKit framework f or building the user interface for this application. You can use your favorite YouTube video for this recipe. By default, we're using a video provided by Apple introducing the new iPad 2 device. How to do it... Following the example from the previous recipe, create a new XHTML file called youtube.html and insert the standard headers for loading the UiUIKit framework. Then add the following CSS inside the <head> section to style the main button: <style type="text/css"> #btn { margin-right: 12px; }</style> Our graphical user interface will be completed by adding the following XHTML code: <div id="header"> <h1>YouTube video</h1></div><h1>Video</h1><p id="btn"> <a href="http://m.youtube.com/watch?v=Z_d6_gbb90I" class="button white">Watch</a></p> After loading the new application on your device, you will see a screen similar to the following screenshot: When the user clicks on our main button, Safari Mobile will go to the web page of the video at YouTube, as shown in the following screenshot: After clicking on the play button, the video will start playing. We can rotate our device for a better aspect ratio, as shown in the following screenshot: How it works... This recipe is pretty simple; we only need to create a link to the desired YouTube video. The most important thing to keep in mind is that we'll use a mobile-specific domain for loading our video to mobile devices. The URL is simply http://m.youtube.com, instead of the regular URL http://www.youtube.com. Posting on your Facebook wall The application developed for this recipe shows how to authenticate with Facebook and how to write a post on your public wall. If everything is successful, an alert box is used to report it to the user. Although there are myriad complex applications with better functionalities that can be built for Facebook, we will focus on simple posting in this recipe. This application will only allow you to post on your wall. For this you need to hardcode your Facebook account for posting. This is to keep the recipe as simple as possible and to get a good understanding of all the complex processes involved in dealing with the OAuth protocol used by Facebook. However, thanks to this open protocol, it is easier to allow secure authentication of APIs from web applications. Also, this recipe requires using a real Internet domain and a server with public access. Thus, you cannot test this application on your local machine. Our application needs to use a server- side language for which we'll use PHP. Currently, it's very easy to find hosting services for PHP applications. You can also find very cheap services for hosting your PHP code. You can find the complete code for this recipe at code/ch10/facebook/ in the code bundle provided on the Packtpub site. Getting ready To bu ild the application for this recipe, you need to have a public server with an Internet domain linked to it. Also, you must install a web server with a PHP interpreter and have the UiUIKit framework ready to use. You need to install the cURL library, which allows PHP to connect and communicate to many different types of servers. Two interesting resources for this issue are: http://www.php.net/manual/en/book.curl.php http://www.php.net/manual/en/curl.setup.php  

Integrating Twitter with Magento Twitter (http://twitter.com) is a micro-blogging service, which allows its users to send short messages to their followers, answering the question "what are you doing now?" After registering a Twitter account, you can begin to follow other Twitter users. When they update their status, you will see it in your timeline of what people you follow say. When you sign up for a Twitter account, it is usually best to sign up as the name of your store—for example, "Cheesy Cheese Store" rather than "RichardCarter", simply because your customers are more likely to search for the name of the store rather than your own name. Tweeting: Ideas for your store's tweets If you look at other businesses on Twitter, you'll see that there are a number of ways to promote your store on Twitter without losing followers by being too "spammy". Some companies give voucher codes to Twitter followers—a good way to entice new customers Others use Twitter to host competitions for free items—a good way to reward existing customers You can also release products to your Twitter followers before releasing them to other customers Displaying your Twitter updates on your Magento store Twitter can be a powerful tool for your store on its own, but you can integrate Twitter with your Magento store to drive existing customers to your Twitter account, which can help to generate repeat customers. There are a few ways Twitter can be used with Magento, the most versatile of which is the LazzyMonks Twitter module. Installing the LazzyMonks Twitter module To install the LazzyMonks module, visit its page on the Magento Commerce web site (http://www.magentocommerce.com/extension/482/lazzymonks-twitter, and retrieve the extension key, after agreeing to the terms and conditions. Log in to your Magento store's administration panel, and open the Magento Connect Manager in the Magento Connect option under the System tab. Once this has loaded, paste the extension key in to the text box next to the Paste extension key to install label, as shown in the following screenshot: This will install the module for you. Return to your Magento store's administration panel, and you will see a Twitter option in the navigation. The View Tweets option allows you to view updates made to your Twitter account. The Post Update option allows you to update Twitter from your store's administration panel. Firstly, you will need to configure the module's settings, which can be found under the Twitter option of the Configuration section of your store's administration panel, under the System tab. The Twitter Login options are of particular interest. Here you will need to enter your Twitter account's username and password. Once this has been saved, you can post a status update to your Twitter account through Magento's administration panel: This then appears on your Twitter account: Your tweets will also be displayed on your store, as a block beneath other content and can be styled with CSS in your Magento theme by addressing div.twitter. Other ways to integrate Twitter with Magento The other way to integrate your Twitter feed with Magento is by embedding Twitter's widgets into your site. To use these, log in to your Twitter account, and go to http://twitter.com/widgets. You can then use the HTML provided within the Magento templates to insert your Twitter updates into your store. Adding your Twitter feed through Magento's CMS Alternatively, you can insert your Twitter account's updates into any page managed through Magento's Content Management System. In Magento's administration panel, select CMS | Manage Pages, and select the page that you want your Twitter stream to appear in. Within your page, simply paste the code that Twitter produces when you select the type of Twitter "badge", which you want to display on your store. Consider creating a new block for your Twitter statuses, so that it can be removed from pages where it is likely to be distracting (for example, the checkout page).

(For more resources related to this topic, see here.) The first step is to set up a Google Docs spreadsheet for the project. Create a new sheet, give it a name (I named mine Power for this project, but you can name it as you wish), and set a title for the columns that we are going to use: Time, Interval, Power, and Energy (that will be calculated from the first two columns), as shown in the following screenshot: We can also calculate the value of the energy using the other measurements. From theory, we know that over a given period of time, energy is power multiplied by time; that is, Energy = Power * Time. However, in our case, power is calculated at regular intervals, and we want to estimate the energy consumption for each of these intervals. In mathematical terms, this means we need to calculate the integral of power as a function of time. We don't have the exact function between time and power as we sample this function at regular time intervals, but we can estimate this integral using a method called the trapezoidal rule. It means that we basically estimate the integral of the function, which is the area below the power curve, by a trapeze. The energy in the C2 cell in the spreadsheet is then given by the formula: Energy= (PowerMeasurement + NextPowerMeasurement)*TimeInverval/2 Concretely, in Google Docs, you will need the formula, D2 = (B2 + B3)*C2/2. The Arduino Yún board will give you the power measurement, and the time interval is given by the value we set in the sketch. However, the time between two measurements can vary from measurement to measurement. This is due to the delay introduced by the network. To solve this issue, we will transmit the exact value along with the power measurement to get a much better estimate of the energy consumption. Then, it's time to build the sketch that we will use for the project. The goal of this sketch is basically to wait for commands that come from the network, to switch the relay on or off, and to send data to the Google Docs spreadsheet at regular intervals to keep track of the energy consumption. We will build the sketch on top of the sketch we built earlier so I will explain which components need to be added. First, you need to include your Temboo credentials using the following line of code: #include "TembooAccount.h" Since we can't continuously measure the power consumption data (the data transmitted would be huge, and we will quickly exceed our monthly access limit for Temboo!), like in the test sketch, we need to measure it at given intervals only. However, at the same time, we need to continuously check whether a command is received from the outside to switch the state of the relay. This is done by setting the correct timings first, as shown in the following code: int server_poll_time = 50; int power_measurement_delay = 10000; int power_measurement_cycles_max = power_measurement_delay/server_ poll_time; The server poll time will be the interval at which we check the incoming connections. The power measurement delay, as you can guess, is the delay at which the power is measured. However, we can't use a simple delay function for this as it will put the entire sketch on hold. What we are going to do instead is to count the number of cycles of the main loop and then trigger a measurement when the right amount of cycles have been reached using a simple if statement. The right amount of cycles is given by the power measurement cycles_max variable. You also need to insert your Google Docs credentials using the following lines of code: const String GOOGLE_USERNAME = "yourGoogleUsername"; const String GOOGLE_PASSWORD = "yourGooglePass"; const String SPREADSHEET_TITLE = "Power"; In the setup() function, you need to start a date process that will keep a track of the measurement date. We want to keep a track of the measurement over several days, so we will transmit the date of the day as well as the time, as shown in the following code: time = millis(); if (!date.running()) { date.begin("date"); date.addParameter("+%D-%T"); date.run(); } In the loop() function of the sketch, we check whether it's time to perform a measurement from the current sensor, as shown in the following line of code: if (power_measurement_cycles > power_measurement_cycles_max); If that's the case, we measure the sensor value, as follows: float sensor_value = getSensorValue(); We also get the exact measurement interval that we will transmit along with the measured power to get a correct estimate of the energy consumption, as follows: measurements_interval = millis() - last_measurement; last_measurement = millis(); We then calculate the effective power from the data we already have. The amplitude of the current is obtained from the sensor measurements. Then, we can get the effective value of the current by dividing this amplitude by the square root of 2. Finally, as we know the effective voltage and that power is current multiplied by voltage, we can calculate the effective power as well, as shown in the following code: // Convert to current amplitude_current=(float)(sensor_value-zero_ sensor)/1024*5/185*1000000; effectivevalue=amplitude_current/1.414; // Calculate power float effective_power = abs(effective_value * effective_voltage/1000); After this, we send the data with the time interval to Google Docs and reset the counter for power measurements, as follows: runAppendRow(measurements_interval,effective_power); power_measurement_cycles = 0; Let's quickly go into the details of this function. It starts by declaring the type of Temboo library we want to use, as follows: TembooChoreo AppendRowChoreo; Start with the following line of code: AppendRowChoreo.begin(); We then need to set the data that concerns your Google account, for example, the username, as follows: AppendRowChoreo.addInput("Username", GOOGLE_USERNAME); The actual formatting of the data is done with the following line of code: data = data + timeString + "," + String(interval) + "," + String(effectiveValue); Here, interval is the time interval between two measurements, and effectiveValue is the value of the measured power that we want to log on to Google Docs. The Choreo is then executed with the following line of code: AppendRowChoreo.run(); Finally, we do this after every 50 milliseconds and get an increment to the power measurement counter each time, as follows: delay(server_poll_time); power_measurement_cycles++; The complete code is available at https://github.com/openhomeautomation/geeky-projects-yun/tree/master/chapter2/energy_log. The code for this part is complete. You can now upload the sketch and after that, open the Google Docs spreadsheet and then just wait until the first measurement arrives. The following screenshot shows the first measurement I got: After a few moments, I got several measurements logged on my Google Docs spreadsheet. I also played a bit with the lamp control by switching it on and off so that we can actually see changes in the measured data. The following screenshot shows the first few measurements: It's good to have some data logged in the spreadsheet, but it is even better to display this data in a graph. I used the built-in plotting capabilities of Google Docs to plot the power consumption over time on a graph, as shown in the following screenshot: Using the same kind of graph, you can also plot the calculated energy consumption data over time, as shown in the following screenshot: From the data you get in this Google Docs spreadsheet, it is also quite easy to get other interesting data. You can, for example, estimate the total energy consumption over time and the price that it will cost you. The first step is to calculate the sum of the energy consumption column using the integrated sum functionality of Google Docs. Then, you have the energy consumption in Joules, but that's not what the electricity company usually charges you for. Instead, they use kWh, which is basically the Joule value divided by 3,600,000. The last thing we need is the price of a single kWh. Of course, this will depend on the country you're living in, but at the time of writing this article, the price in the USA was approximately $0.16 per kWh. To get the total price, you then just need to multiply the total energy consumption in kWh with the price per kWh. This is the result with the data I recorded. Of course, as I only took a short sample of data, it cost me nearly nothing in the end, as shown in the following screenshot: You can also estimate the on/off time of the device you are measuring. For this purpose, I simply added an additional column next to Energy named On/Off. I simply used the formula =IF(C2<2;0;1). It means that if the power is less than 2W, we count it as an off state; otherwise, we count it as an on state. I didn't set the condition to 0W to count it as an off state because of the small fluctuations over time from the current sensor. Then, when you have this data about the different on/off states, it's quite simple to count the number of occurrences of each state, for example, on states, using =COUNTIF(E:E,"1"). I applied these formulas in my Google Docs spreadsheet, and the following screenshot is the result with the sample data I recorded: It is also very convenient to represent this data in a graph. For this, I used a pie chart, which I believe is the most adaptable graph for this kind of data. The following screenshot is what I got with my measurements: With the preceding kind of chart, you can compare the usage of a given lamp from day to day, for example, to know whether you have left the lights on when you are not there. Summary In this article, we learned to send data to Google docs, measure the energy consumption, and store this data to the Web. Resources for Article: Further resources on this subject: Home Security by BeagleBone [Article] Playing with Max 6 Framework [Article] Our First Project – A Basic Thermometer [Article]

This article is the first part of a series of articles, please refer to Part 2  for learning how to add a Chatbot to the portfolio website you create in this article!Creating a personal portfolio is a great way for showcasing your skills and accomplishments as a developer or designer. Does that sound like a lot of work? Well… it doesn’t have to be. We can use ChatGPT to generate code snippets and obtain a lot of guidance throughout the process. This way you can build an impressive portfolio website with minimal effort.Here’s what you can do in around 10 prompts: Fig 1: Homepage Fig 2: Portfolio PageFig 3: Contact Page Not bad, right? And it even contains some features: The search functionality works and filters projects based on what you are typing.The testimonials and projects are not hard-coded, but dynamically populated with JavaScript (but not connected to a backend with a database, so for the purpose of this article, they are hard-coded there).Of course, this personal portfolio would need more content. And you could definitely use ChatGPT to generate a bit more content and descriptions for it. You should be adding in some professional pictures and images to complete your portfolio. However, this is a project that you can do in half a day to a day using ChatGPT. The prompt magic The prompts used to create the personal portfolio follow a few best practices that I’ll discuss later. In the overview below, asking to continue after cutting off a response is not included. If you’d like to see all the responses and the versions of the personal portfolio after each step, you can check out this GitHub repo: https://github.com/BrightBoost/brightboost-portfolio-chatgpt Here is the first prompt:I'm making a portfolio site, using HTML/CSS/JS. Can you help me write the initial framework for it?  I need a home page, in which I will introduce me, my passions, aspirations and study background. I also need a call-to-action directing visitors to explore my work.Then I need a portfolio page, in which I showcase my best projects. This will include case studies, and testimonials highlighting my contributions and accomplishments.Finally I need a contact page, which is a minimalist form and social media links in order to facilitate professional connections. And the second:That seems to be close to what I want. However, could you split these into 3 files? One for each page.The third, as you can see, a lot of the heavy lifting in terms of content is done in the prompt here:Can you populate the pages with the following information?Name: Olivia MartinezStudy: Olivia recently graduated from the University of California, Berkeley with a Bachelor's degree in Computer Science.Aspirations: Olivia aspires to become a full-stack software engineer at a tech startup that focuses on environmental sustainability. She hopes to contribute her skills to develop innovative solutions for pressing environmental issues.Hobbies: In her free time, Olivia enjoys hiking, painting, and exploring the latest advancements in machine learning.Example Open Source Projects:- GreenRoute: A web application that optimizes travel routes based on carbon emissions.- EcoClicker: A browser extension that encourages users to reduce their digital carbon footprint.Additional Personal Details: Olivia has been an active volunteer at her local recycling center, where she has helped develop a digital platform to streamline recycling processes. This is what it looked like after this prompt:Fig 4: Homepage after initial promptsFig 5: Portfolio page after promptFig 6: Contact Page after promptThe fourth prompt was quite a challenge and it required going back and forward a bit and testing it until it was good. It was tempting to just modify it, but ChatGPT was supposed to create it here and it did eventually:Can you help me modify the following snippet? ```html      <h2>Portfolio</h2>      <div class="project">        <h3>GreenRoute</h3>        <p>A web application that optimizes travel routes based            on carbon emissions.</p>        <a href="#" class="project-link">View Case Study</a>        <div class="testimonials">          <p>Testimonial 1</p>          <p>Testimonial 2</p>        </div>      </div>       <div class="project">        <h3>EcoClicker</h3>        <p>A browser extension that encourages users to reduce            their digital carbon footprint.</p>        <a href="#" class="project-link">View Case Study</a>        <div class="testimonials">          <p>Testimonial 1</p>          <p>Testimonial 2</p>        </div>      </div>    ``` I'm not satisfied with the look. Could you make the following changes: - Each project is displayed in a card.- The project link looks like a button, in the bottom right.- The title is underlined, and a bit larger.- The page shows 2 columns of cards. Fig 7: Home page after refined promptingAnd here’s the fifth: I need to make sure that footer is always at the bottom of the page, can you provide a CSS snippet to make that work?This also needed second attempt because it wasn’t working. Don’t just say that it doesn’t work, but be specific:It doesn't seem to work. The page only uses about 50% of the screen, so the footer is still in the middle. After this, it looks like: Fig 8: Homepage after footer promptsThis is where things really got cool, but this needed a few tweaks in terms of output. Here was the first prompt to add JavaScript: I'd like to make the portfolio a bit more extendable. Can you write some JavaScript code that generates the portfolio page using an array of objects? For now just put the content directly in code. I forgot a few classes, so let’s prompt again: This works, but you've excluded the classes used in the CSS. As a reminder, this is how a single item should look:** code of the prompt omitted And after this it was good: It seems the 2 column layout is gone. I think this:```html<section id="portfolio"><div class="container" id="portfolio-container"></div></section>```Should contain an element with the class `project-grid` somewhere, which should create a grid. Can you modify the snippet? The last part was on the search bar, which required this prompt:I'd like to add a search bar to the portfolio page. It must search for the text in the title and body. I only want to look for the exact text. After each character it should update the list, filtering out any project that does not match the search text. Then there should be a button to clear the search bar, and show all projects. Can you add this to the JavaScript file? And that’s it! Of course, there are many ways to do this, but this is one way of how you can use ChatGPT to create a personal portfolio. Let’s see some best practices for your ChatGPT prompts, to help you with using it to create your personal portfolio.Best practices for ChatGPT prompts There are some best practices I figured out when working with ChatGPT. Let’s go over them before seeing the prompts used for the personal portfolio.Be specific and clear: Make sure your prompt leaves little room for interpretation. For example, the prompt:Help me with a grid layout.Is not going to help you as much as:For this piece of HTML containing bootstrap cards provide a CSS snippet for a responsive 3-column grid layout with a 20px gap between columns: ** insert your HTML snippet here **Include relevant context and background information: Give the AI enough information to understand the problem or task and help you to its best ability. Don’t ask:How do I convert a date string to a Date object?But ask:  I have a JSON object with date and value properties. How do I convert the date property to a JavaScript Date object?Ask one question at a time: Keep your prompts focused and avoid asking multiple questions in one prompt.Make sure ChatGPT completes its answer before asking the next question: Sometimes it cuts off the result. You can ask it to continue and it will. That’s harder when you’re further down the line.Test the result after every step: Related to the previous tip, but make sure to test the result after every step. This way you can provide feedback on the outcome and it can easily adjust still. Step? Yes! Break down big projects into smaller tasks: Divide your project into manageable steps, and ask the AI to complete each task separately.Bonus tip: You can even ask ChatGPT for help on how to break your project into smaller tasks and make these tasks very detailed. Then go ahead and ask it to do one task at a time.The good news is these tips are actually great interaction tips with humans as well! I bet you’d like to see some of the prompts used to create the personal portfolio, so let’s dive in. Author BioMaaike van Putten is an experienced software developer and Pluralsight, LinkedIn Learning, Udemy, and Bright Boost instructor. She has a passion for software development and helping others get to the next level in their career.You can follow Maaike on:LinkedInTraining Courses

Adding form splitters Commonly used forms like Sales orders or Projects in Dynamics AX have multiple grids. Normally, one grid is in the upper section and another one is in the bottom section of the form. Sometimes grids are placed next to each other. The size of the data in each grid may vary, and that's why most of the forms with multiple grids have splitters in the middle so users can resize both grids at once by dragging the splitter with the help of a mouse. It is a good practice to add splitters to newly created forms. Although Microsoft developers did a good job by adding splitters to most of the multi-grid forms, there is still at least one that has not got it. It is the Account reconciliation form in the Bank module, which is one of the most commonly used forms. It can be opened from Bank | Bank Account Details, Functions | Account reconciliation button, and then the Transactions button. In the following screenshot, you can see that the size of the bottom grid cannot be changed: In this recipe, we will demonstrate the usage of splitters by resolving this situation. We will add a form splitter in the middle of the two grids in the mentioned form. It will allow users to define the sizes of both grids to make sure that the data is displayed optimally. How to do it... Open the BankReconciliation form in AOT, and create a new Group at the very top of the form's design with the following properties:   Property Value Name Top AutoDeclaration Yes FrameType None Width Column width   Move the AllReconciled, Balances, and Tab controls into the newly created group. Create a new Group right below the Top group with properties: Property Value Name Splitter AutoDeclaration Yes Width Column width Height 5 FrameType Raised 3D BackgroundColor Window background HideIfEmpty No AlignChild No Add the following line of code to the bottom of the form's class declaration: SysFormSplitter_Y fs; Add the following line of code to the bottom of the form's init(): fs = new SysFormSplitter_Y(Splitter, Top, element); Override three methods in the Splitter group with the following code: public int mouseDown( int _x, int _y, int _button, boolean _ctrl, boolean _shift) { return fs.mouseDown(_x, _y, _button, _ctrl, _shift); } public int mouseMove( int _x, int _y, int _button, boolean _ctrl, boolean _shift) { return fs.mouseMove(_x, _y, _button, _ctrl, _shift); } public int mouseUp( int _x, int _y, int _button, boolean _ctrl, boolean _shift) { return fs.mouseUp(_x, _y, _button, _ctrl, _shift); } Change the following properties of the existing BankTransTypeGroup group: Property Value Top Auto Width Column width Height Column height Change the following property of the exiting TypeSums grid located in BankTransTypeGroup group: Property Value Height Column height In AOT the Modified BankReconciliation form should look like the following screenshot: Now, to test the results, open Bank | Bank Account Details, select any bank account, click Functions | Account reconciliation, choose an existing or create a new account statement, and click the Transactions button. Notice that now the form has a nice splitter in the middle, which makes the form look better and allows defining the size of each grid: How it works... Normally a splitter is placed between two form groups. In this recipe, to follow that rule, we need to adjust the BankReconciliation form's design. The filter AllReconciled, the group Balances and the tab Tab are moved to a new group called Top. We do not want this new group to be visible to user, so we set FrameType to None. Setting AutoDeclaration to Yes allows us to access this object from X++ code. And finally, we make this group automatically expand in the horizontal direction by setting its Width to Column width. At this stage, visual form layout did not change, but now we have the upper group ready. The BankTransTypeGroup group could be used as a bottom group with slight changes. We change its Top behavior to Auto and make it fully expandable in the horizontal and vertical directions. The Height of the grid inside this group also has to be changed to Column height in order to fill all the vertical space. In the middle of those two groups, we add a splitter. The splitter is nothing else but another group, which looks like a splitter. In order to achieve that, we set Height to 5, FrameType to Raised 3D, and BackgroundColor to Windows background. This group does not hold any other controls inside. Therefore, in order to make it visible, we have to set the property HideIfEmpty to No. The value No of the property AlignChild makes the splitter begin on the very left side of the form and the Column width value of the property Width forces the splitter to automatically fill the form's width. Mouse events are handled by the SysFormSplitter_Y application class. After it has been declared in the form's class declaration, we create the actual object in the form's init(). We pass the name of the splitter control, the name of the top group and the form itself as arguments when creating it. A fully working splitter requires three mouse event handlers. It is implemented by overriding the mouseMove(), mouseDown(), and mouseUp() methods in the splitter group control. All arguments are passed to the respective member methods of the SysFormSplitter_Y class which does all the job. In this way, horizontal splitters can be easily added to any form. The Dynamics AX application also contains nice examples about splitters, which can be found in AOT in the Tutorial_Form_Split form. Vertical splitters can also be added to forms using a very similar approach. For this, we need to use another application class called SysFormSplitter_X. Creating modal forms During my trainings and working with Dynamics AX users, I noticed that people who are not familiar with computers and software tend to get lost among open application windows. The same could be applied to Dynamics AX. I experienced many times when a user opened one form, clicked some button to open another one, and then went back to the first one without closing the second one. Sometimes this happens intentionally, sometimes—not, but the result is that the second form is hidden behind the first one and the user starts wondering why it is not possible to close or edit the first form. Such issues could be easily solved by making the child form a modal window. In other words, the second form always stays on top of the fi rst one until closed. In this recipe, we will do exactly that. As an example, we will make the Create sales order form a modal window. How to do it... Open the SalesCreateOrder form in AOT, and set its Design property: Property Value WindowType Popup To test, open Accounts receivable | Sales Order Details, and start creating a new order. Notice that now the sales order creation form always stays on top of the Sales order form: How it works... Dynamics AX form design has a WindowType property, which is set to Standard by default. In order to make a form behave as a modal window, we have to change it to Popup. Such forms will always stay on top of the parent form. There's more... We already know that some of the Dynamics AX forms are created dynamically using the Dialog class. If we look deeper into the code, we could find that the Dialog class actually creates a runtime Dynamics AX form. That means we can apply the same principle, i.e. change the relevant form's design property. The following code could be added to the Dialog object and would do the job: dialog.dialogForm().buildDesign().windowType( FormWindowType::Popup); We get a reference to the form's design, by first using dialogForm() of the Dialog object to get a reference to the DialogForm object, and then we call buildDesign() on the latter object. Then, we set the design's property by calling its windowType() with an argument FormWindowType::Popup. Changing common form appearance In every single multi-company Dynamics AX project, in order to prevent user mistakes, I was asked to add functionality that allows setting the background color of every form per company. By doing that, users clearly see in which company account they are at the moment and can easily work within multiple companies at the same time. In this recipe, we will modify SysSetupFormRun class to change the background color for every form in Dynamics AX. How to do it... Open SysSetupFormRun in AOT, and override its run() with the following code: public void run() {; super(); this.design().colorScheme(FormColorScheme::RGB); this.design().backgroundColor(WinAPI::RGB2int(255,0,0)); } To test the results, open any Dynamics AX form, for example, General ledger | Chart of Accounts Details and notice how the background color is changed to red: How it works... SysSetupFormRun is the application class that is called by the system every time a user runs a form. The best place to add our custom code is to override the run() method and place it under the super(). We use this.design() to get a reference to the form's design. By calling colorScheme() and backgroundColor(), we set the color scheme to red/green/blue and the color code to red. We use WinAPI::RGB2int() to transform the human-readable red/green/blue code into the numeric color code. There's more... This recipe showed a very basic principle of how to change the common appearance of all forms with few lines of code. You noticed that the color in this recipe does not fi ll all areas of the form, which does not make the form look nice. An alternative to this could be to dynamically add a colored rectangle or something similar to the top of the form. The possibilities are endless here. New controls like input fields, buttons, menu items, and others could also be added to all forms dynamically using this class. But do not overdo as it may impact system performance. Storing last form values Dynamics AX has a very useful feature, which allows saving the latest user choices per user per form. This feature is implemented across a number of standard reports, periodic jobs, and other objects, which require user input. When developing a new functionality for Dynamics AX, I always try to keep that practice. One of the frequently used areas is custom filters for grid-based forms. Although, Dynamics AX allows users to use standard filtering for any grid, in practice sometimes it is not very useful, especially when the user requires something specific. In this recipe, we will see how to store the latest user filter selections. To make it as simple as possible, we will use existing filters on the General journal form, which can be opened from General ledger | Journals | General journal. This form contains two filters—Show and Show user-created only. Show allows displaying journals by their posting status and Show user-created only toggles between all journals and the currently logged user's journals. How to do it... Find the LedgerJournalTable form in AOT, and add the following code to the bottom of its class declaration: AllOpenPosted showStatus; NoYes showCurrentUser; #define.CurrentVersion(1) #localmacro.CurrentList showStatus, showCurrentUser #endmacro Create these additional form methods: public void initParmDefault() {; showStatus = AllOpenPosted::Open; showCurrentUser = true; } public container pack() { return [#CurrentVersion,#CurrentList]; } public boolean unpack(container packedClass) { int version = RunBase::getVersion(packedClass); ; switch (version) { case #CurrentVersion: [version,#CurrentList] = packedClass; return true; default: return false; } return false; } public identifiername lastValueDesignName() { return element.args().menuItemName(); } public identifiername lastValueElementName() { return this.name(); } public UtilElementType lastValueType() { return UtilElementType::Form; } public userId lastValueUserId() { return curuserid(); } public dataAreaId lastValueDataAreaId() { return curext(); } xSysLastValue::getLast(this); AllOpenPostedField.selection(showStatus); ShowUserCreatedOnly.value(showCurrentUser); journalFormTable.designSelectionChangeAllOpenPosted(); journalFormTable.designSelectionChangeShowUserCreateOnly(); And the following code to the bottom of the form's close(): showStatus = AllOpenPostedField.selection(); showCurrentUser = ShowUserCreatedOnly.value(); xSysLastValue::saveLast(this); Now to test the form, open General ledger | Journals | General journal, change filter values, close it, and run again. The latest filter selections should stay: How it works... First of all, we define some variables. We will store the journal posting status filter value in showStatus and the current user filter value in showCurrentUser. The macro #CurrentList is used to define a list of variables that we are going to store. Currently, we have two variables. The macro #CurrentVersion defines a version of saved values. In other words, it says that the variables defined by the #CurrentList, which will be stored in system cache later, can be addressed using the number 1. Normally, when implementing last value saving for the first time for particular object, #CurrentVersion is set to 1. Later on, if we decide to add new values or change existing ones, we have to change the value of #CurrentVersion, normally increasing it by 1. This ensures that the system addresses the correct list of variables in the cache and does not break existing functionality. The initParmDefault()method specifies default values if nothing is found in the system cache. Normally, this happens if we run a form for the first time, we change #CurrentVersion or clean the cache. Later, this method is called automatically by the xSysLastValue object. The methods pack() and unpack() are responsible for formatting a storage container from variables and extracting variables from a storage container respectively. In our case, pack() returns a container consisting of three values: version number, posting status, and current user toggle. Those values will be sent to the system cache after the form is closed. During an opening of the form, the xSysLastValue object uses unpack() to extract values from the stored container. It checks the container version from cache first, and if it matches the current version number, then the values from the cache are considered correct and are assigned to the form variables. A combination of lastValueDesignName(), lastValueElementName(), lastValueType(), and lastValueDataAreaId() return values form unique string representing saved values. This ensures that different users can store last values for different objects without overriding each other's values in cache. The lastValueDesignName() method is meant to return the name of the object's current design in cases where the object can have several designs. In this recipe, there is only one design, so instead of leaving it empty, I used it for a slightly different purpose. The same LedgerJournalTable AOT form can represent different user forms like Ledger journal, Periodic journal, Vendor payment journal, and so on depending on the location from which it was opened. To ensure that the user's latest choices are saved correctly, we included the opening menu item name as part of the unique string. The last two pieces of code need to be added to the bottom of the form's run() and close(). In the run() method, xSysLastValue::getLast(this) retrieves saved user values from cache and assigns them to the form's variables. The next two lines assign the same values to the respective form controls. designSelectionChangeAllOpenPosted() and designSelectionChangeShowUserCreateOnly() execute a form query to apply updated filters. Although both of those methods currently perform exactly the same action, we keep both for the future in case this functionality is updated. Code lines in close() are responsible for assigning user selections to variables and saving them to cache by calling xSysLastValue::saveLast(this).