Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-elon-musks-neuralink-unveils-a-sewing-machine-like-robot-to-control-computers-via-the-brain
Sugandha Lahoti
17 Jul 2019
8 min read
Save for later

Elon Musk's Neuralink unveils a “sewing machine-like” robot to control computers via the brain

Sugandha Lahoti
17 Jul 2019
8 min read
After two years of being super-secretive about their work, Neuralink, Elon’s Musk’s neurotechnology company, has finally presented their progress in brain-computer interface technology. The Livestream which was uploaded on YouTube showcases a “sewing machine-like” robot that can implant ultrathin threads deep into the brain giving people the ability to control computers and smartphones using their thoughts. For its brain-computer interface tech, the company has received $158 million in funding and has 90 employees. Note: All images are taken from Neuralink Livestream video unless stated otherwise. Elon Musk opened the presentation talking about the primary aim of Neuralink which is to use brain-computer interface tech to understand and treat brain disorders, preserve and enhance the brain, and ultimately and this may sound weird, “achieve a symbiosis with artificial intelligence”. He added, “This is not a mandatory thing. It is a thing you can choose to have if you want. This is something that I think will be really important on a civilization-level scale.” Neuralink wants to build, record from and selectively stimulate as many neurons as possible across diverse brain areas. They have three goals: Increase by orders of magnitude, the number of neurons you can read from and write to in safe, long-lasting ways. At each stage, produce devices that serve critical unmet medical needs of patients. Make inserting a computer connection into your brain as safe and painless as LASIK eye surgery. The robot that they have built was designed to be completely wireless, with a  practical bandwidth that is usable at home and lasts for a long time. Their system has an N1 sensor, which is an 8mm wide, 4mm tall cylinder having 1024 electrodes. It consists of a thin film, which has threads. The threads are placed using thin needles, into the brain by a robotic system in a manner akin to a sewing machine avoiding blood vessels. The robot peels off the threads one by one from the N1 Sensor and places it in the brain. A needle would grab each thread by a small loop and then is inserted into the brain by the robot. The robot is under the supervision of a human neurosurgeon who lays out where the threads are placed. The actual needle which the robot uses is 24 microns. The process puts a 2mm incision near the human ear, which is dilated to 8mm. The threads A robot implants threads using a needle For the first patients, the Neuralink team is looking at four sensors which will be connected via very small wires under the scalp to an inductive coil behind the ear. This is encased in a wearable device that they call the ‘Link’ which contains a Bluetooth radio and a battery. They will be controlled through an iPhone app. Source: NYT Neuralink/MetaLab iPhone app The goal is to drill four 8mm holes into paralyzed patients’ skulls and insert implants that will give them the ability to control computers and smartphones using their thoughts. For the first product, they are focusing on giving patients the ability to control their mobile device, and then redirect the output from their phone to a keyboard or a mouse. The company will seek U.S. Food and Drug Administration approval and is aspiring to target first-in-human clinical study by 2020. They will use it for treating upper cervical spinal cord injury. They’re expecting those patients to get four 1024 channel sensors, one each in the primary motor cortex, supplementary motor area, premotor cortex and closed-loop feedback into the primary somatosensory cortex. As reported by Bloomberg who got a pre-media briefing, Neuralink said it has performed at least 19 surgeries on animals with its robots and successfully placed the wires, which it calls “threads,” about 87% of the time. They used a lab rat and implanted a USB-C port in its head. A wire attached to the port transmitted its thoughts to a nearby computer where a software recorded and analyzed its brain activity, measuring the strength of brain spikes. The amount of data being gathered from a lab rat was about 10 times greater than what today’s most powerful sensors can collect. The flexibility of the Neuralink threads would be an advance, said Terry Sejnowski, the Francis Crick Professor at the Salk Institute for Biological Studies, in La Jolla, Calif to the New York Times. However, he noted that the Neuralink researchers still needed to prove that the insulation of their threads could survive for long periods in a brain’s environment, which has a salt solution that deteriorates many plastics. Musk's bizarre attempts to revolutionalize the world are far from reality Elon Musk is known for his dramatic promises and showmanship as much as he is for his eccentric projects. But how far they are grounded in reality is another thing. In May he successfully launched his mammoth space mission, Starlink sending 60 communications satellites to the orbit which will eventually be part of a single constellation providing high-speed internet to the globe. However, the satellites were launched after postponing it two times to “update satellite software”. Not just that,  three of the 60 satellites have lost contact with ground control teams, a SpaceX spokesperson said on June 28. Experts are already worried about how the Starlink constellation will contribute to the space debris problem. Currently, there are 2,000 operational satellites in orbit around Earth, according to the latest figures from the European Space Agency, and the completed Starlink constellation will drastically add to that number. Observers had also noticed some Starlink satellites had not initiated orbit raising after being released. Musk’s much-anticipated Hyperloop (first publicly mentioned in 2012) was supposed to shuttle passengers at near-supersonic speeds via pods traveling in a long, underground tunnel. But it was soon reduced to a car in a very small tunnel. When they unveiled the underground tunnel to the media in California last year in December, reporters climbed into electric cars made by Musk’s Tesla and were treated to a 40 mph ride along a bumpy path. Here as well there have been public concerns regarding its impact on public infrastructure and the environment. The biggest questions surrounding hyperloop’s environmental impact are its effect on carbon dioxide emissions, the effect of infrastructure on ecosystems, and the environmental footprint of the materials used to build it. Other concerns include noise pollution and how to repurpose hyperloop tubes and tunnels at the end of their lifespan. Researchers from Tencent Keen Security Lab criticized Tesla’s self-driving car software, publishing a report detailing their successful attacks on Tesla firmware. It includes remote control over the steering and an adversarial example attack on the autopilot that confuses the car into driving into oncoming traffic lane. Musk had also made promises to have a fully self-driving car for Tesla by 2020 which caused a lot of activity in the stock markets. But most are skeptical about this claim as well. Whether Elon Musk’s AI symbiotic visions will come in existence in the foreseeable future is questionable. Neuralink's long-term goals are characteristically unrealistic, considering not much is known about the human brain; cognitive functions and their representation as brain signals are still an area where much further research is required. While Musk’s projects are known for their technical excellence, History shows a lack of thought into the broader consequences and cost of such innovations such as the ethical concerns, environmental and societal impacts. Neuralink’s implant is also prone to invading one’s privacy as it will be storing sensitive medical information of a patient. There is also the likelihood of it violating one’s constitutional rights such as freedom of speech, expression among others. What does it mean to live in a world where one’s thoughts are constantly monitored and not truly one’s own? Then, because this is an implant what if the electrodes malfunction and send wrong signals to the brain. Who will be accountable in such scenarios? Although the FDA will be probing into such questions, these are some questions any responsible company should ask of itself proactively while developing life-altering products or services. These are equally important aspects that are worthy of stage time in a product launch. Regardless, Musk’s bold claims and dramatic representations are sure to gain the attention of investors and enthusiasts for now. Elon Musk reveals big plans with Neuralink SpaceX shares new information on Starlink after the successful launch of 60 satellites What Elon Musk can teach us about Futurism & Technology Forecasting
Read more
  • 0
  • 0
  • 14693

article-image-using-cloud-applications-and-containers
Xavier Bruhiere
10 Nov 2015
7 min read
Save for later

Using Cloud Applications and Containers

Xavier Bruhiere
10 Nov 2015
7 min read
We can find a certain comfort while developing an application on our local computer. We debug logs in real time. We know the exact location of everything, for we probably started it by ourselves. Make it work, make it right, make it fast - Kent Beck Optimization is the root of all devil - Donald Knuth So hey, we hack around until interesting results pop up (ok that's a bit exaggerated). The point is, when hitting the production server our code will sail a much different sea. And a much more hostile one. So, how to connect to third party resources ? How do you get a clear picture of what is really happening under the hood ? In this post we will try to answer those questions with existing tools. We won't discuss continuous integration or complex orchestration. Instead, we will focus on what it takes to wrap a typical program to make it run as a public service. A sample application Before diving into the real problem, we need some code to throw on remote servers. Our sample application below exposes a random key/value store over http. // app.js // use redis for data storage var Redis = require('ioredis'); // and express to expose a RESTFul API var express = require('express'); var app = express(); // connecting to redis server var redis = new Redis({ host: process.env.REDIS_HOST || '127.0.0.1', port: process.env.REDIS_PORT || 6379 }); // store random float at the given path app.post('/:key', function (req, res) { var key = req.params.key var value = Math.random(); console.log('storing', value,'at', key) res.json({set: redis.set(key, value)}); }); // retrieve the value at the given path app.get('/:key', function (req, res) { console.log('fetching value at ', req.params.key); redis.get(req.params.key).then(function(err, result) { res.json({ result: result || err }); }) }); var server = app.listen(3000, function () { var host = server.address().address; var port = server.address().port; console.log('Example app listening at http://%s:%s', host, port); }); And we define the following package.json and Dockerfile. { "name": "sample-app", "version": "0.1.0", "scripts": { "start": "node app.js" }, "dependencies": { "express": "^4.12.4", "ioredis": "^1.3.6", }, "devDependencies": {} } # Given a correct package.json, those two lines alone will properly install and run our code FROM node:0.12-onbuild # application's default port EXPOSE 3000 A Dockerfile ? Yeah, here is a first step toward cloud computation under control. Packing our code and its dependencies into a container will allow us to ship and launch the application with a few reproducible commands. # download official redis image docker pull redis # cd to the root directory of the app and build the container docker build -t article/sample . # assuming we are logged in to hub.docker.com, upload the resulting image for future deployment docker push article/sample Enough for the preparation, time to actually run the code. Service Discovery The server code needs a connection to redis. We can't hardcode it because host and port are likely to change under different deployments. Fortunately The Twelve-Factor App provides us with an elegant solution. The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; Indeed, this strategy integrates smoothly with an infrastructure composed of containers. docker run --detach --name redis redis # 7c5b7ff0b3f95e412fc7bee4677e1c5a22e9077d68ad19c48444d55d5f683f79 # fetch redis container virtual ip export REDIS_HOST=$(docker inspect -f '{{ .NetworkSettings.IPAddress }}' redis) # note : we don't specify REDIS_PORT as the redis container listens on the default port (6379) docker run -it --rm --name sample --env REDIS_HOST=$REDIS_HOST article/sample # > sample-app@0.1.0 start /usr/src/app # > node app.js # Example app listening at http://:::3000 In another terminal, we can check everything is working as expected. export SAMPLE_HOST=$(docker inspect -f '{{ .NetworkSettings.IPAddress }}' sample)) curl -X POST $SAMPLE_HOST:3000/test # {"set":{"isFulfilled":false,"isRejected":false}} curl -X GET $SAMPLE_HOST:3000/test # {"result":"0.5807915225159377"} We didn't precise any network informations but even so, containers can communicate. This method is widely used and projects like etcd or consul let us automate the whole process. Monitoring Performances can be a critical consideration for end-user experience or infrastructure costs. We should be able to identify bottlenecks or abnormal activities and once again, we will take advantage of containers and open source projects. Without modifying the running server, let's launch three new components to build a generic monitoring infrastructure. Influxdb is a fast time series database where we will store containers metrics. Since we properly defined the application into two single-purpose containers, it will give us an interesting overview of what's going on. # default parameters export INFLUXDB_PORT=8086 export INFLUXDB_USER=root export INFLUXDB_PASS=root export INFLUXDB_NAME=cadvisor # Start database backend docker run --detach --name influxdb --publish 8083:8083 --publish $INFLUXDB_PORT:8086 --expose 8090 --expose 8099 --env PRE_CREATE_DB=$INFLUXDB_NAME tutum/influxdb export INFLUXDB_HOST=$(docker inspect -f '{{ .NetworkSettings.IPAddress }}' influxdb) cadvisor Analyzes resource usage and performance characteristics of running containers. The command flags will instruct it how to use the database above to store metrics. docker run --detach --name cadvisor --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 google/cadvisor:latest --storage_driver=influxdb --storage_driver_user=$INFLUXDB_USER --storage_driver_password=$INFLUXDB_PASS --storage_driver_host=$INFLUXDB_HOST:$INFLUXDB_PORT --log_dir=/ # A live dashboard is available at $CADVISOR_HOST:8080/containers # We can also point the brower to $INFLUXDB_HOST:8083, with credentials above, to inspect containers data. # Query example: # > list series # > select time,memory_usage from stats where container_name='cadvisor' limit 1000 # More infos: https://github.com/google/cadvisor/blob/master/storage/influxdb/influxdb.go Grafana is a feature rich metrics dashboard and graph editor for Graphite, InfluxDB and OpenTSB. From its web interface, we will query the database and graph the metrics cadvisor collected and stored. docker run --detach --name grafana -p 8000:80 -e INFLUXDB_HOST=$INFLUXDB_HOST -e INFLUXDB_PORT=$INFLUXDB_PORT -e INFLUXDB_NAME=$INFLUXDB_NAME -e INFLUXDB_USER=$INFLUXDB_USER -e INFLUXDB_PASS=$INFLUXDB_PASS -e INFLUXDB_IS_GRAFANADB=true tutum/grafana # Get login infos generated docker logs grafana  Now we can head to localhost:8000 and build a custom dashboard to monitor the server. I won't repeat the comprehensive documentation but here is a query example: # note: cadvisor stores metrics in series named 'stats' select difference(cpu_cumulative_usage) where container_name='cadvisor' group by time 60s Grafana's autocompletion feature shows us what we can track : cpu, memory and network usage among other metrics. We all love screenshots and dashboards so here is a final reward for our hard work. Conclusion Development best practices and a good understanding of powerful tools gave us a rigorous workflow to launch applications with confidence. To sum up: Containers bundle code and requirements for flexible deployment and execution isolation. Environment stores third party services informations, giving developers a predictable and robust solution to read them. InfluxDB + Cadvisor + Grafana feature a complete monitoring solution independently of the project implementation. We fullfilled our expections but there's room for improvements. As mentioned, service discovery could be automated, but we also omitted how to manage logs. There are many discussions around this complex subject and we can expect shortly new improvements in our toolbox. About the author Xavier Bruhiere is the CEO of Hive Tech. He contributes to many community projects, including Occulus Rift, Myo, Docker and Leap Motion. In his spare time he enjoys playing tennis, the violin and the guitar. You can reach him at @XavierBruhiere.
Read more
  • 0
  • 0
  • 14686

article-image-classify-emails-using-deep-neural-networks-generating-tf-idf
Savia Lobo
21 Feb 2018
9 min read
Save for later

How to classify emails using deep neural networks after generating TF-IDF

Savia Lobo
21 Feb 2018
9 min read
[box type="note" align="" class="" width=""]This article is an excerpt taken from the book Natural Language Processing with Python Cookbook written by Krishna Bhavsar, Naresh Kumar, and Pratap Dangeti. This book will teach you how to efficiently use NLTK and implement text classification, identify parts of speech, tag words, and more. You will also learn how to analyze sentence structures and master lexical analysis, syntactic and semantic analysis, pragmatic analysis, and application of deep learning techniques.[/box] In this article, you will learn how to use deep neural networks to classify emails into one of the 20 pre-trained categories based on the words present in each email. This is a simple model to start with understanding the subject of deep learning and its applications on NLP. Getting ready The 20 newsgroups dataset from scikit-learn have been utilized to illustrate the concept. Number of observations/emails considered for analysis are 18,846 (train observations - 11,314 and test observations - 7,532) and its corresponding classes/categories are 20, which are shown in the following: >>> from sklearn.datasets import fetch_20newsgroups >>> newsgroups_train = fetch_20newsgroups(subset='train') >>> newsgroups_test = fetch_20newsgroups(subset='test') >>> x_train = newsgroups_train.data >>> x_test = newsgroups_test.data >>> y_train = newsgroups_train.target >>> y_test = newsgroups_test.target >>> print ("List of all 20 categories:") >>> print (newsgroups_train.target_names) >>> print ("n") >>> print ("Sample Email:") >>> print (x_train[0]) >>> print ("Sample Target Category:") >>> print (y_train[0]) >>> print (newsgroups_train.target_names[y_train[0]]) In the following screenshot, a sample first data observation and target class category has been shown. From the first observation or email we can infer that the email is talking about a two-door sports car, which we can classify manually into autos category which is 8. Note: Target value is 7 due to the indexing starts from 0), which is validating our understanding with actual target class 7. How to do it… Using NLP techniques, we have pre-processed the data for obtaining finalized word vectors to map with final outcomes spam or ham. Major steps involved are:    Pre-processing.    Removal of punctuations.    Word tokenization.    Converting words into lowercase.    Stop word removal.    Keeping words of length of at least 3.    Stemming words.    POS tagging.    Lemmatization of words: TF-IDF vector conversion. Deep learning model training and testing. Model evaluation and results discussion. How it works... The NLTK package has been utilized for all the pre-processing steps, as it consists of all the necessary NLP functionality under one single roof: # Used for pre-processing data >>> import nltk >>> from nltk.corpus import stopwords >>> from nltk.stem import WordNetLemmatizer >>> import string >>> import pandas as pd >>> from nltk import pos_tag >>> from nltk.stem import PorterStemmer The function written (pre-processing) consists of all the steps for convenience. However, we will be explaining all the steps in each section: >>> def preprocessing(text): The following line of the code splits the word and checks each character to see if it contains any standard punctuations, if so it will be replaced with a blank or else it just don't replace with blank: ... text2 = " ".join("".join([" " if ch in string.punctuation else ch for ch in text]).split()) The following code tokenizes the sentences into words based on whitespaces and puts them together as a list for applying further steps: ... tokens = [word for sent in nltk.sent_tokenize(text2) for word in nltk.word_tokenize(sent)] Converting all the cases (upper, lower and proper) into lower case reduces duplicates in corpus: ... tokens = [word.lower() for word in tokens] As mentioned earlier, Stop words are the words that do not carry much of weight in understanding the sentence; they are used for connecting words and so on. We have removed them with the following line of code: ... stopwds = stopwords.words('english') ... tokens = [token for token in tokens if token not in stopwds] Keeping only the words with length greater than 3 in the following code for removing small words which hardly consists of much of a meaning to carry; ... tokens = [word for word in tokens if len(word)>=3] Stemming applied on the words using Porter stemmer which stems the extra suffixes from the words: ... stemmer = PorterStemmer() ... tokens = [stemmer.stem(word) for word in tokens] POS tagging is a prerequisite for lemmatization, based on whether word is noun or verb or and so on. it will reduce it to the root word ... tagged_corpus = pos_tag(tokens) pos_tag function returns the part of speed in four formats for Noun and six formats for verb. NN - (noun, common, singular), NNP - (noun, proper, singular), NNPS - (noun, proper, plural), NNS - (noun, common, plural), VB - (verb, base form), VBD - (verb, past tense), VBG - (verb, present participle), VBN - (verb, past participle), VBP - (verb, present tense, not 3rd person singular), VBZ - (verb, present tense, third person singular) ... Noun_tags = ['NN','NNP','NNPS','NNS'] ... Verb_tags = ['VB','VBD','VBG','VBN','VBP','VBZ'] ... lemmatizer = WordNetLemmatizer() The following function, prat_lemmatize, has been created only for the reasons of mismatch between the pos_tag function and intake values of lemmatize function. If the tag for any word falls under the respective noun or verb tags category, n or v will be applied accordingly in lemmatize function: ... def prat_lemmatize(token,tag): ...      if tag in Noun_tags: ...          return lemmatizer.lemmatize(token,'n') ...      elif tag in Verb_tags: ...          return lemmatizer.lemmatize(token,'v') ...      else: ...          return lemmatizer.lemmatize(token,'n') After performing tokenization and applied all the various operations, we need to join it back to form stings and the following function performs the same: ... pre_proc_text =   " ".join([prat_lemmatize(token,tag) for token,tag in tagged_corpus]) ... return pre_proc_text Applying pre-processing on train and test data: >>> x_train_preprocessed = [] >>> for i in x_train: ... x_train_preprocessed.append(preprocessing(i)) >>> x_test_preprocessed = [] >>> for i in x_test: ... x_test_preprocessed.append(preprocessing(i)) # building TFIDF vectorizer >>> from sklearn.feature_extraction.text import TfidfVectorizer >>> vectorizer = TfidfVectorizer(min_df=2, ngram_range=(1, 2), stop_words='english', max_features= 10000,strip_accents='unicode', norm='l2') >>> x_train_2 = vectorizer.fit_transform(x_train_preprocessed).todense() >>> x_test_2 = vectorizer.transform(x_test_preprocessed).todense() After the pre-processing step has been completed, processed TF-IDF vectors have to be sent to the following deep learning code: # Deep Learning modules >>> import numpy as np >>> from keras.models import Sequential >>> from keras.layers.core import Dense, Dropout, Activation >>> from keras.optimizers import Adadelta,Adam,RMSprop >>> from keras.utils import np_utils The following image produces the output after firing up the preceding Keras code. Keras has been installed on Theano, which eventually works on Python. A GPU with 6 GB memory has been installed with additional libraries (CuDNN and CNMeM) for four to five times faster execution, with a choking of around 20% memory; hence only 80% memory out of 6 GB is available; The following code explains the central part of the deep learning model. The code is self- explanatory, with the number of classes considered 20, batch size 64, and number of epochs to train, 20: # Definition hyper parameters >>> np.random.seed(1337) >>> nb_classes = 20 >>> batch_size = 64 >>> nb_epochs = 20 The following code converts the 20 categories into one-hot encoding vectors in which 20 columns are created and the values against the respective classes are given as 1. All other classes are given as 0: >>> Y_train = np_utils.to_categorical(y_train, nb_classes) In the following building blocks of Keras code, three hidden layers (1000, 500, and 50 neurons in each layer respectively) are used, with dropout as 50% for each layer with Adam as an optimizer: #Deep Layer Model building in Keras #del model >>> model = Sequential() >>> model.add(Dense(1000,input_shape= (10000,))) >>> model.add(Activation('relu')) >>> model.add(Dropout(0.5)) >>> model.add(Dense(500)) >>> model.add(Activation('relu')) >>> model.add(Dropout(0.5)) >>> model.add(Dense(50)) >>> model.add(Activation('relu')) >>> model.add(Dropout(0.5)) >>> model.add(Dense(nb_classes)) >>> model.add(Activation('softmax')) >>> model.compile(loss='categorical_crossentropy', optimizer='adam') >>> print (model.summary()) The architecture is shown as follows and describes the flow of the data from a start of 10,000 as input. Then there are 1000, 500, 50, and 20 neurons to classify the given email into one of the 20 categories: The model is trained as per the given metrics: # Model Training >>> model.fit(x_train_2, Y_train, batch_size=batch_size, epochs=nb_epochs,verbose=1) The model has been fitted with 20 epochs, in which each epoch took about 2 seconds. The loss has been minimized from 1.9281 to 0.0241. By using CPU hardware, the time required for training each epoch may increase as a GPU massively parallelizes the computation with thousands of threads/cores: Finally, predictions are made on the train and test datasets to determine the accuracy, precision, and recall values: #Model Prediction >>> y_train_predclass = model.predict_classes(x_train_2,batch_size=batch_size) >>> y_test_predclass = model.predict_classes(x_test_2,batch_size=batch_size) >>> from sklearn.metrics import accuracy_score,classification_report >>> print ("nnDeep Neural Network - Train accuracy:"),(round(accuracy_score( y_train, y_train_predclass),3)) >>> print ("nDeep Neural Network - Test accuracy:"),(round(accuracy_score( y_test,y_test_predclass),3)) >>> print ("nDeep Neural Network - Train Classification Report") >>> print (classification_report(y_train,y_train_predclass)) >>> print ("nDeep Neural Network - Test Classification Report") >>> print (classification_report(y_test,y_test_predclass)) It appears that the classifier is giving a good 99.9% accuracy on the train dataset and 80.7% on the test dataset. We learned the classification of emails using DNNs(Deep Neural Networks) after generating TF-IDF. If you found this post useful, do check out this book Natural Language Processing with Python Cookbook  to further analyze sentence structures and application of various deep learning techniques.    
Read more
  • 0
  • 0
  • 14680

article-image-how-to-build-a-neural-network-to-fill-the-missing-part-of-a-handwritten-digit-using-gans-tutorial
Melisha Dsouza
05 Jan 2019
18 min read
Save for later

How to build a neural network to fill the missing part of a handwritten digit using GANs [Tutorial]

Melisha Dsouza
05 Jan 2019
18 min read
GANs are neural networks used in unsupervised learning that generate synthetic data given certain input data. GAN's have two components: a generator and a discriminator. A generator generates new instances of an object and the discriminator determines whether the new instance belongs to the actual dataset. A generative learn how the data is generated i.e. the structure of the data, in order to categorize it. This allows the system to generate samples with similar statistical properties. Discriminative models will learn the relation between the data and the label associated with the data. The discriminative model will categorize the input data without knowing how the data is generated. GAN exploits the concept behind both the models to get a better network architecture. This tutorial on GAN's will help you build a neural network that fills in the missing part of a handwritten digit. This tutorial will cover how to build an MNIST digit classifier and simulate a dataset of handwritten digits with sections of the handwritten numbers missing. Next, users will learn using the MNIST classifier to predict on noised/masked MNIST digits dataset (simulated dataset) and implement GAN to generate back the missing regions of the digit. This tutorial will also cover using the MNIST classifier to predict on the generated digits from GAN and finally compare performance between masked data and generated data. This tutorial is an excerpt from a book written by  Matthew Lamons, Rahul Kumar, Abhishek Nagaraja titled Python Deep Learning Projects. This book will help users develop their own deep learning systems in a straightforward way and in an efficient way. The book has projects developed using complex deep learning projects in the field of computational linguistics and computer vision to help users master the subject. All of the Python files and Jupyter Notebook files for this tutorial can be found at GitHub. In this tutorial, we will be using the Keras deep learning library. Importing all of the dependencies We will be using numpy, matplotlib, keras, tensorflow, and the tqdm package in this exercise. Here, TensorFlow is used as the backend for Keras. You can install these packages with pip. For the MNIST data, we will be using the dataset available in the keras module with a simple import: import numpy as np import random import matplotlib.pyplot as plt %matplotlib inline from tqdm import tqdm from keras.layers import Input, Conv2D from keras.layers import AveragePooling2D, BatchNormalization from keras.layers import UpSampling2D, Flatten, Activation from keras.models import Model, Sequential from keras.layers.core import Dense, Dropout from keras.layers.advanced_activations import LeakyReLU from keras.optimizers import Adam from keras import backend as k from keras.datasets import mnist It is important that you set seed for reproducibility: # set seed for reproducibility seed_val = 9000 np.random.seed(seed_val) random.seed(seed_val) Exploring the data We will load the MNIST data into our session from the keras module with mnist.load_data(). After doing so, we will print the shape and the size of the dataset, as well as the number of classes and unique labels in the dataset: (X_train, y_train), (X_test, y_test) = mnist.load_data() print('Size of the training_set: ', X_train.shape) print('Size of the test_set: ', X_test.shape) print('Shape of each image: ', X_train[0].shape) print('Total number of classes: ', len(np.unique(y_train))) print('Unique class labels: ', np.unique(y_train)) We have a dataset with 10 different classes and 60,000 images, with each image having a shape of 28*28 and each class having 6,000 images. Let's plot and see what the handwritten images look like: # Plot of 9 random images for i in range(0, 9): plt.subplot(331+i) # plot of 3 rows and 3 columns plt.axis('off') # turn off axis plt.imshow(X_train[i], cmap='gray') # gray scale The output is as follows: Let's plot a handwritten digit from each class: # plotting image from each class fig=plt.figure(figsize=(8, 4)) columns = 5 rows = 2 for i in range(0, rows*columns): fig.add_subplot(rows, columns, i+1) plt.title(str(i)) # label plt.axis('off') # turn off axis plt.imshow(X_train[np.where(y_train==i)][0], cmap='gray') # gray scale plt.show() The output is as follows: Look at the maximum and the minimum pixel value in the dataset: print('Maximum pixel value in the training_set: ', np.max(X_train)) print('Minimum pixel value in the training_set: ', np.min(X_train)) The output is as follows: Preparing the data Type conversion, centering, scaling, and reshaping are some of the pre-processing we will implement in this tutorial. Type conversion, centering and scaling Set the type to np.float32. For centering, we subtract the dataset by 127.5. The values in the dataset will now range between -127.5 to 127.5. For scaling, we divide the centered dataset by half of the maximum pixel value in the dataset, that is, 255/2. This will result in a dataset with values ranging between -1 and 1: # Converting integer values to float types X_train = X_train.astype(np.float32) X_test = X_test.astype(np.float32) # Scaling and centering X_train = (X_train - 127.5) / 127.5 X_test = (X_test - 127.5)/ 127.5 print('Maximum pixel value in the training_set after Centering and Scaling: ', np.max(X_train)) print('Minimum pixel value in the training_set after Centering and Scaling: ', np.min(X_train)) Let's define a function to rescale the pixel values of the scaled image to range between 0 and 255: # Rescale the pixel values (0 and 255) def upscale(image): return (image*127.5 + 127.5).astype(np.uint8) # Lets see if this works z = upscale(X_train[0]) print('Maximum pixel value after upscaling scaled image: ',np.max(z)) print('Maximum pixel value after upscaling scaled image: ',np.min(z)) A plot of 9 centered and scaled images after upscaling: for i in range(0, 9): plt.subplot(331+i) # plot of 3 rows and 3 columns plt.axis('off') # turn off axis plt.imshow(upscale(X_train[i]), cmap='gray') # gray scale The output is as follows: Masking/inserting noise For the needs of this project, we need to simulate a dataset of incomplete digits. So, let's write a function to mask small regions in the original image to form the noised dataset. The idea is to mask an 8*8 region of the image with the top-left corner of the mask falling between the 9th and 13th pixel (between index 8 and 12) along both the x and y axis of the image. This is to make sure that we are always masking around the center part of the image: def noising(image): array = np.array(image) i = random.choice(range(8,12)) # x coordinate for the top left corner of the mask j = random.choice(range(8,12)) # y coordinate for the top left corner of the mask array[i:i+8, j:j+8]=-1.0 # setting the pixels in the masked region to -1 return array noised_train_data = np.array([*map(noising, X_train)]) noised_test_data = np.array([*map(noising, X_test)]) print('Noised train data Shape/Dimension : ', noised_train_data.shape) print('Noised test data Shape/Dimension : ', noised_train_data.shape) A plot of 9 scaled noised images after upscaling: # Plot of 9 scaled noised images after upscaling for i in range(0, 9): plt.subplot(331+i) # plot of 3 rows and 3 columns plt.axis('off') # turn off axis plt.imshow(upscale(noised_train_data[i]), cmap='gray') # gray scale The output is as follows: Reshaping Reshape the original dataset and the noised dataset to a shape of 60000*28*28*1. This is important since the 2D convolutions expect to receive images of a shape of 28*28*1: # Reshaping the training data X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], X_train.shape[2], 1) print('Size/Shape of the original training set: ', X_train.shape) # Reshaping the noised training data noised_train_data = noised_train_data.reshape(noised_train_data.shape[0], noised_train_data.shape[1], noised_train_data.shape[2], 1) print('Size/Shape of the noised training set: ', noised_train_data.shape) # Reshaping the testing data X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], X_test.shape[2], 1) print('Size/Shape of the original test set: ', X_test.shape) # Reshaping the noised testing data noised_test_data = noised_test_data.reshape(noised_test_data.shape[0], noised_test_data.shape[1], noised_test_data.shape[2], 1) print('Size/Shape of the noised test set: ', noised_test_data.shape) MNIST classifier To start off with modeling, let's build a simple convolutional neural network (CNN) digit classifier. The first layer is a convolution layer that has 32 filters of a shape of 3*3, with relu activation and Dropout as the regularizer. The second layer is a convolution layer that has 64 filters of a shape of 3*3, with relu activation and Dropout as the regularizer. The third layer is a convolution layer that has 128 filters of a shape of 3*3, with relu activation and Dropout as the regularizer, which is finally flattened. The fourth layer is a Dense layer of 1024 neurons with relu activation. The final layer is a Dense layer with 10 neurons corresponding to the 10 classes in the MNIST dataset, and the activation used here is softmax, batch_size is set to 128, the optimizer used is adam, and validation_split is set to 0.2. This means that 20% of the training set will be used as the validation set: # input image shape input_shape = (28,28,1) def train_mnist(input_shape, X_train, y_train): model = Sequential() model.add(Conv2D(32, (3, 3), strides=2, padding='same', input_shape=input_shape)) model.add(Activation('relu')) model.add(Dropout(0.2)) model.add(Conv2D(64, (3, 3), strides=2, padding='same')) model.add(Activation('relu')) model.add(Dropout(0.2)) model.add(Conv2D(128, (3, 3), padding='same')) model.add(Activation('relu')) model.add(Dropout(0.2)) model.add(Flatten()) model.add(Dense(1024, activation = 'relu')) model.add(Dense(10, activation='softmax')) model.compile(loss = 'sparse_categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy']) model.fit(X_train, y_train, batch_size = 128, epochs = 3, validation_split=0.2, verbose = 1 ) return model mnist_model = train_mnist(input_shape, X_train, y_train) The output is as follows: Use the built CNN digit classifier on the masked images to get a measure of its performance on digits that are missing small sections: # prediction on the masked images pred_labels = mnist_model.predict_classes(noised_test_data) print('The model model accuracy on the masked images is:',np.mean(pred_labels==y_test)*100) On the masked images, the CNN digit classifier is 74.9% accurate. It might be slightly different when you run it, but it will still be very close. Defining hyperparameters for GAN The following are some of the hyperparameters defined that we will be using throughout the code and are totally configurable: # Smoothing value smooth_real = 0.9 # Number of epochs epochs = 5 # Batchsize batch_size = 128 # Optimizer for the generator optimizer_g = Adam(lr=0.0002, beta_1=0.5) # Optimizer for the discriminator optimizer_d = Adam(lr=0.0004, beta_1=0.5) # Shape of the input image input_shape = (28,28,1) Building the GAN model components With the idea that the final GAN model will be able to fill in the part of the image that is missing (masked), let's define the generator. You can understand how to define the generator, discriminator, and DCGAN by referring to our book. Training GAN We've built the components of the GAN.  Let's train the model in the next steps! Plotting the training – part 1 During each epoch, the following function plots 9 generated images. For comparison, it will also plot the corresponding 9 original target images and 9 noised input images. We need to use the upscale function we've defined when plotting to make sure the images are scaled to range between 0 and 255, so that you do not encounter issues when plotting: def generated_images_plot(original, noised_data, generator): print('NOISED') for i in range(9): plt.subplot(331 + i) plt.axis('off') plt.imshow(upscale(np.squeeze(noised_data[i])), cmap='gray') # upscale for plotting plt.show() print('GENERATED') for i in range(9): pred = generator.predict(noised_data[i:i+1], verbose=0) plt.subplot(331 + i) plt.axis('off') plt.imshow(upscale(np.squeeze(pred[0])), cmap='gray') # upscale to avoid plotting errors plt.show() print('ORIGINAL') for i in range(9): plt.subplot(331 + i) plt.axis('off') plt.imshow(upscale(np.squeeze(original[i])), cmap='gray') # upscale for plotting plt.show() The output of this function is as follows: Plotting the training – part 2 Let's define another function that plots the images generated during each epoch. To reflect the difference, we will also include the original and the masked/noised images in the plot. The top row contains the original images, the middle row contains the masked images, and the bottom row contains the generated images. The plot has 12 rows with the sequence, row 1 - original, row 2 - masked, row3 - generated, row 4 - original, row5 - masked,..., row 12 - generated. Let's take a look at the code for the same: def plot_generated_images_combined(original, noised_data, generator): rows, cols = 4, 12 num = rows * cols image_size = 28 generated_images = generator.predict(noised_data[0:num]) imgs = np.concatenate([original[0:num], noised_data[0:num], generated_images]) imgs = imgs.reshape((rows * 3, cols, image_size, image_size)) imgs = np.vstack(np.split(imgs, rows, axis=1)) imgs = imgs.reshape((rows * 3, -1, image_size, image_size)) imgs = np.vstack([np.hstack(i) for i in imgs]) imgs = upscale(imgs) plt.figure(figsize=(8,16)) plt.axis('off') plt.title('Original Images: top rows, ' 'Corrupted Input: middle rows, ' 'Generated Images: bottom rows') plt.imshow(imgs, cmap='gray') plt.show() The output is as follows: Training loop Now we are at the most important part of the code; the part where all of the functions we previously defined will be used. The following are the steps: Load the generator by calling the img_generator() function. Load the discriminator by calling the img_discriminator() function and compile it with the binary cross-entropy loss and optimizer as optimizer_d, which we have defined under the hyperparameters section. Feed the generator and the discriminator to the dcgan() function and compile it with the binary cross-entropy loss and optimizer as optimizer_g, which we have defined under the hyperparameters section. Create a new batch of original images and masked images. Generate new fake images by feeding the batch of masked images to the generator. Concatenate the original and generated images so that the first 128 images are all original and the next 128 images are all fake. It is important that you do not shuffle the data here, otherwise it will be hard to train. Label the generated images as 0 and original images as 0.9 instead of 1. This is one-sided label smoothing on the original images. The reason for using label smoothing is to make the network resilient to adversarial examples. It's called one-sided because we are smoothing labels only for the real images. Set discriminator.trainable to True to enable training of the discriminator and feed this set of 256 images and their corresponding labels to the discriminator for classification. Now, set discriminator.trainable to False and feed a new batch of 128 masked images labeled as 1 to the GAN (DCGAN) for classification. It is important to set discriminator.trainable to False to make sure the discriminator is not getting trained while training the generator. Repeat steps 4 through 7 for the desired number of epochs. We have placed the plot_generated_images_combined() function and the generated_images_plot() function to  get a plot generated by both functions after the first iteration in the first epoch and after the end of each epoch. Feel free to place these plot functions according to the frequency of plots you need displayed: def train(X_train, noised_train_data, input_shape, smooth_real, epochs, batch_size, optimizer_g, optimizer_d): # define two empty lists to store the discriminator # and the generator losses discriminator_losses = [] generator_losses = [] # Number of iteration possible with batches of size 128 iterations = X_train.shape[0] // batch_size # Load the generator and the discriminator generator = img_generator(input_shape) discriminator = img_discriminator(input_shape) # Compile the discriminator with binary_crossentropy loss discriminator.compile(loss='binary_crossentropy',optimizer=optimizer_d) # Feed the generator and the discriminator to the function dcgan # to form the DCGAN architecture gan = dcgan(discriminator, generator, input_shape) # Compile the DCGAN with binary_crossentropy loss gan.compile(loss='binary_crossentropy', optimizer=optimizer_g) for i in range(epochs): print ('Epoch %d' % (i+1)) # Use tqdm to get an estimate of time remaining for j in tqdm(range(1, iterations+1)): # batch of original images (batch = batchsize) original = X_train[np.random.randint(0, X_train.shape[0], size=batch_size)] # batch of noised images (batch = batchsize) noise = noised_train_data[np.random.randint(0, noised_train_data.shape[0], size=batch_size)] # Generate fake images generated_images = generator.predict(noise) # Labels for generated data dis_lab = np.zeros(2*batch_size) # data for discriminator dis_train = np.concatenate([original, generated_images]) # label smoothing for original images dis_lab[:batch_size] = smooth_real # Train discriminator on original images discriminator.trainable = True discriminator_loss = discriminator.train_on_batch(dis_train, dis_lab) # save the losses discriminator_losses.append(discriminator_loss) # Train generator gen_lab = np.ones(batch_size) discriminator.trainable = False sample_indices = np.random.randint(0, X_train.shape[0], size=batch_size) original = X_train[sample_indices] noise = noised_train_data[sample_indices] generator_loss = gan.train_on_batch(noise, gen_lab) # save the losses generator_losses.append(generator_loss) if i == 0 and j == 1: print('Iteration - %d', j) generated_images_plot(original, noise, generator) plot_generated_images_combined(original, noise, generator) print("Discriminator Loss: ", discriminator_loss,\ ", Adversarial Loss: ", generator_loss) # training plot 1 generated_images_plot(original, noise, generator) # training plot 2 plot_generated_images_combined(original, noise, generator) # plot the training losses plt.figure() plt.plot(range(len(discriminator_losses)), discriminator_losses, color='red', label='Discriminator loss') plt.plot(range(len(generator_losses)), generator_losses, color='blue', label='Adversarial loss') plt.title('Discriminator and Adversarial loss') plt.xlabel('Iterations') plt.ylabel('Loss (Adversarial/Discriminator)') plt.legend() plt.show() return generator generator = train(X_train, noised_train_data, input_shape, smooth_real, epochs, batch_size, optimizer_g, optimizer_d) The output is as follows:  Generated images plotted with training plots at the end of the first iteration of epoch 1              Generated images plotted with training plots at the end of epoch 2              Generated images plotted with training plots at the end of epoch 5      Plot of the discriminator and adversarial loss during training Predictions CNN classifier predictions on the noised and generated images We will call the generator on the masked MNIST test data to generate images, that is, fill in the missing part of the digits: # restore missing parts of the digit with the generator gen_imgs_test = generator.predict(noised_test_data) Then, we will pass the generated MNIST digits to the digit classifier we have modeled already: # predict on the restored/generated digits gen_pred_lab = mnist_model.predict_classes(gen_imgs_test) print('The model model accuracy on the generated images is:',np.mean(gen_pred_lab==y_test)*100) The MNIST CNN classifier is 87.82% accurate on the generated data. The following is a plot showing 10 generated images by the generator, the actual label of the generated image, and the label predicted by the digit classifier after processing the generated image: # plot of 10 generated images and their predicted label fig=plt.figure(figsize=(8, 4)) plt.title('Generated Images') plt.axis('off') columns = 5 rows = 2 for i in range(0, rows*columns): fig.add_subplot(rows, columns, i+1) plt.title('Act: %d, Pred: %d'%(gen_pred_lab[i],y_test[i])) # label plt.axis('off') # turn off axis plt.imshow(upscale(np.squeeze(gen_imgs_test[i])), cmap='gray') # gray scale plt.show() The output is as follows: The Jupyter Notebook code files for the preceding DCGAN MNIST inpainting can be found at GitHub. Use the Jupyter Notebook code files for the DCGAN Fashion MNIST inpainting can be found. Summary We built a deep convolution GAN in Keras on handwritten MNIST digits and understood the function of the generator and the discriminator component of the GAN. We defined key hyperparameters, as well as, in some places, reasoned with why we used what we did. Finally, we tested the GAN's performance on unseen data and determined that we succeeded in achieving our goals. To understand insightful projects to master deep learning and neural network architectures using Python and Keras, check out this book  Python Deep Learning Projects. Getting started with Web Scraping using Python [Tutorial] Google researchers introduce JAX: A TensorFlow-like framework for generating high-performance code from Python and NumPy machine learning programs Google releases Magenta studio beta, an open source python machine learning library for music artists
Read more
  • 0
  • 0
  • 14675

article-image-creating-custom-hud
Packt
29 Apr 2013
5 min read
Save for later

Creating a Custom HUD

Packt
29 Apr 2013
5 min read
(For more resources related to this topic, see here.) Mission Briefing In this project we will be creating a HUD that can be used within a Medieval RPG and that will fit nicely into the provided Epic Citadel map, making use of Scaleform and ActionScript 3.0 using Adobe Flash CS6. As usual, we will be following a simple step—by—step process from beginning to end to complete the project. Here is the outline of our tasks: Setting up Flash Creating our HUD Importing Flash files into UDK Setting up Flash Our first step will be setting up Flash in order for us to create our HUD. In order to do this, we must first install the Scaleform Launcher. Prepare for Lift Off At this point, I will assume that you have run Adobe Flash CS6 at least once beforehand. If not, you can skip this section to where we actually import the .swf file into UDK. Alternatively, you can try to use some other way to create a Flash animation, such as FlashDevelop, Flash Builder, or SlickEdit; but that will have to be done on your own. Engage Thrusters The first step will be to install the Scaleform Launcher. The launcher will make it very easy for us to test our Flash content using the GFX hardware—accelerated Flash Player, which is what UDK will use to play it. Let's get started. Open up Adobe Flash CS6 Professional. Once the program starts up, open up Adobe Extension Manager by going to Help | Manage Extensions.... You may see the menu say Performing configuration tasks, please wait.... This is normal; just wait for it to bring up the menu as shown in the following screenshot: Click on the Install option from the top menu on the right—hand side of the screen. In the file browser, locate the path of your UDK installation and then go into the BinariesGFxCLICK Tools folder. Once there, select the ScaleformExtensions.mxp file and then select OK. When the agreement comes up, press the Accept button; then select whether you want the program to be installed for just you or everyone on your computer. If Flash is currently running, you should get a window popping up telling you that the program will not be ready until you restart the program. Close the manager and restart the program. With your reopened version of Flash start up the Scaleform Launcher by clicking on Window | Other Panels | Scaleform Launcher. At this point you should see the Scaleform Launcher panel come up as shown in the following screenshot: At this point all of the options are grayed out as it doesn't know how to access the GFx player, so let's set that up now. Click on the + button to add a new profile. In the profile name section, type in GFXMediaPlayer. Next, we need to reference the GFx player. Click on the + button in the player EXE section. Go to your UDK directory, BinariesGFx, and then select GFxMediaPlayerD3d9.exe. It will then ask you to give a name for the Player Name field with the value already filled in; just hit the OK button. UDK by default uses DirectX 9 for rendering. However, since GDC 2011, it has been possible for users to use DirectX 11. If your project is using 11, feel free to check out http://udn.epicgames.com/Three/DirectX11Rendering.html and use DX11. In order to test our game, we will need to hit the button that says Test with: GFxMediaPlayerD3d9 as shown in the following screenshot: If you know the resolution in which you want your final game to be, you can set up multiple profiles to preview how your UI will look at a specific resolution. For example, if you'd like to see something at a resolution of 960 x 720, you can do so by altering the command params field after %SWF PATH% to include the text —res 960:720. Now that we have the player loaded, we need to install the CLIK library for our usage. Go to the Preferences menu by selecting Edit | Preferences. Click on the ActionScript tab and then click on the ActionScript 3.0 Settings... button. From there, add a new entry to our Source path section by clicking on the + button. After that, click on the folder icon to browse to the folder we want. Add an additional path to our CLIK directory in the file explorer by first going to your UDK installation directory and then going to DevelopmentFlashAS3CLIK. Click on the OK button and drag—and—drop the newly created Scaleform Launcher to the bottom—right corner of the interface. Objective Complete — Mini Debriefing Alright, Flash is now set up for us to work with Scaleform within it, which for all intents and purposes is probably the hardest part about working with Scaleform. Now that we have taken care of it, let's get started on the HUD! As long as you have administrator access to your computer, these settings should be set for whenever you are working with Flash. However, if you do not, you will have to run through all of these settings every time you want to work on Scaleform projects.
Read more
  • 0
  • 0
  • 14668

article-image-magento-theme-development
Packt
24 Feb 2016
7 min read
Save for later

Magento Theme Development

Packt
24 Feb 2016
7 min read
In this article by Fernando J. Miguel, author of the book Magento 2 Development Essentials, we will learn the basics of theme development. Magento can be customized as per your needs because it is based on the Zend framework, adopting the Model-View-Controller (MVC) architecture as a software design pattern. When planning to create your own theme, the Magento theme process flow becomes a subject that needs to be carefully studied. Let's focus on the concepts that help you create your own theme. (For more resources related to this topic, see here.) The Magento base theme The Magento Community Edition (CE) version 1.9 comes with a new theme named rwd that implements the Responsive Web Design (RWD) practices. Magento CE's responsive theme uses a number of new technologies as follows: Sass/Compass: This is a CSS precompiler that provides a reusable CSS that can even be organized well. jQuery: This is used for customization of JavaScript in the responsive theme. jQuery operates in the noConflict() mode, so it doesn't conflict with Magento's existing JavaScript library. Basically, the folders that contain this theme are as follows: app/design/frontend/rwd skin/frontend/rwd The following image represents the folder structure: As you can see, all the files of the rwd theme are included in the app/design/frontend and skin/frontend folders: app/design/frontend: This folder stores all the .phtml visual files and .xml configurations files of all the themes. skin/frontend: This folder stores all JavaScript, CSS, and image files from their respective app/design/frontend themes folders. Inside these folders, you can see another important folder called base. The rwd theme uses some base theme features to be functional. How is it possible? Logically, Magento has distinct folders for every theme, but Magento is very smart to reuse code. Magento takes advantage of fall-back system. Let's check how it works. The fall-back system The frontend of Magento allows the designers to create new themes based on the basic theme, reusing the main code without changing its structure. The fall-back system allows us to create only the files that are necessary for the customization. To create the customization files, we have the following options: Create a new theme directory and write the entire new code Copy the files from the base theme and edit them as you wish The second option could be more productive for study purposes. You will learn basic structure by exercising the code edit. For example, let's say you want to change the header.phtml file. You can copy the header.html file from the app/design/frontend/base/default/template/page/html path to the app/design/frontend/custom_package/custom_theme/template/page/html path. In this example, if you activate your custom_theme on Magento admin panel, your custom_theme inherits all the structure from base theme, and applies your custom header.phtml on the theme. Magento packages and design themes Magento has the option to create design packages and themes as you saw on the previous example of custom_theme. This is a smart functionality because on same packages you can create more than one theme. Now, let's take a deep look at the main folders that manage the theme structure in Magento. The app/design structure In the app/design structure, we have the following folders: The folder details are as follows: adminhtml: In this folder, Magento keeps all the layout configuration files and .phtml structure of admin area. frontend: In this folder, Magento keeps all the theme's folders and their respective .phtml structure of site frontend. install: This folder stores all the files of installation Magento screen. The layout folder Let's take a look at the rwd theme folder: As you can see, the rwd is a theme folder and has a template folder called default. In Magento, you can create as many template folders as you wish. The layout folders allow you to define the structure of the Magento pages through the XML files. The layout XML files has the power to manage the behavior of your .phtml file: you can incorporate CSS or JavaScript to be loaded on specific pages. Every page on Magento is defined by a handle. A handle is a reference name that Magento uses to refer to a particular page. For example, the <cms_page> handle is used to control the layout of the pages in your Magento. In Magento, we have two main type of handles: Default handles: These manage the whole site Non-default handles: These manage specific parts of the site In the rwd theme, the .xml files are located in app/design/frontend/rwd/default/layout. Let's take a look at an .xml layout file example: This piece of code belongs to the page.xml layout file. We can see the <default> handle defining the .css and .js files that will be loaded on the page. The page.xml file has the same name as its respective folder in app/design/frontend/rwd/default/template/page. This is an internal Magento control. Please keep this in mind: Magento works with a predefined naming file pattern. Keeping this in your mind can avoid unnecessary errors. The template folder The template folder, taking rwd as a reference, is located at app/design/frontend/rwd/default/template. Every subdirectory of template controls a specific page of Magento. The template files are the .phtml files, a mix of HTML and PHP, and they are the layout structure files. Let's take a look at a page/1column.phtml example: The locale folder The locale folder has all the specific translation of the theme. Let's imagine that you want to create a specific translation file for the rwd theme. You can create a locale file at app/design/frontend/rwd/locale/en_US/translate.csv. The locale folder structure basically has a folder of the language (en_US), and always has the translate.csv filename. The app/locale folder in Magento is the main translation folder. You can take a look at it to better understand. But the locale folder inside the theme folder has priority in Magento loading. For example, if you want to create a Brazilian version of the theme, you have to duplicate the translate.csv file from app/design/frontend/rwd/locale/en_US/ to app/design/frontend/rwd/locale/pt_BR/. This will be very useful to those who use the theme and will have to translate it in the future. Creating new entries in translate If you want to create a new entry in your translate.csv, first of all put this code in your PHTML file: <?php echo $this->__('Translate test'); ?> In CSV file, you can put the translation in this format: 'Translate test', 'Translate test'. The SKIN structure The skin folder basically has the css and js files and images of the theme, and is located in skin/frontend/rwd/default. Remember that Magento has a filename/folder naming pattern. The skin folder named rwd will work with rwd theme folder. If Magento has rwd as a main theme and is looking for an image that is not in the skin folder, Magento will search this image in skin/base folder. Remember also that Magento has a fall-back system. It is keeping its search in the main themes folder to find the correct file. Take advantage of this! CMS blocks and pages Magento has a flexible theme system. Beyond Magento code customization, the admin can create blocks and content on Magento admin panel. CMS (Content Management System) pages and blocks on Magento give you the power to embed HTML code in your page. Summary In this article, we covered the basic concepts of Magento theme. These may be used to change the display of the website or its functionality. These themes are interchangeable with Magento installations. Resources for Article: Further resources on this subject: Preparing and Configuring Your Magento Website [article] Introducing Magento Extension Development [article] Installing Magento [article]
Read more
  • 0
  • 0
  • 14663
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-making-app-react-and-material-design
Soham Kamani
21 Mar 2016
7 min read
Save for later

Making an App with React and Material Design

Soham Kamani
21 Mar 2016
7 min read
There has been much progression in the hybrid app development space, and also in React.js. Currently, almost all hybrid apps use cordova to build and run web applications on their platform of choice. Although learning React can be a bit of a steep curve, the benefit you get is that you are forced to make your code more modular, and this leads to huge long-term gains. This is great for developing applications for the browser, but when it comes to developing mobile apps, most web apps fall short because they fail to create the "native" experience that so many users know and love. Implementing these features on your own (through playing around with CSS and JavaScript) may work, but it's a huge pain for even something as simple as a material-design-oriented button. Fortunately, there is a library of react components to help us out with getting the look and feel of material design in our web application, which can then be ported to a mobile to get a native look and feel. This post will take you through all the steps required to build a mobile app with react and then port it to your phone using cordova. Prerequisites and dependencies Globally, you will require cordova, which can be installed by executing this line: npm install -g cordova Now that this is done, you should make a new directory for your project and set up a build environment to use es6 and jsx. Currently, webpack is the most popular build system for react, but if that's not according to your taste, there are many more build systems out there. Once you have your project folder set up, install react as well as all the other libraries you would be needing: npm init npm install --save react react-dom material-ui react-tap-event-plugin Making your app Once we're done, the app should look something like this:   If you just want to get your hands dirty, you can find the source files here. Like all web applications, your app will start with an index.html file: <html> <head> <title>My Mobile App</title> </head> <body> <div id="app-node"> </div> <script src="bundle.js" ></script> </body> </html> Yup, that's it. If you are using webpack, your CSS will be included in the bundle.js file itself, so there's no need to put "style" tags either. This is the only HTML you will need for your application. Next, let's take a look at index.js, the entry point to the application code: //index.js import React from 'react'; import ReactDOM from 'react-dom'; import App from './app.jsx'; const node = document.getElementById('app-node'); ReactDOM.render( <App/>, node ); What this does is grab the main App component and attach it to the app-node DOM node. Drilling down further, let's look at the app.jsx file: //app.jsx'use strict';import React from 'react';import AppBar from 'material-ui/lib/app-bar';import MyTabs from './my-tabs.jsx';let App = React.createClass({ render : function(){ return ( <div> <AppBar title="My App" /> <MyTabs /> </div> ); }});module.exports = App; Following react's philosophy of structuring our code, we can roughly break our app down into two parts: The title bar The tabs below The title bar is more straightforward and directly fetched from the material-ui library. All we have to do is supply a "title" property to the AppBar component. MyTabs is another component that we have made, put in a different file because of the complexity: 'use strict';import React from 'react';import Tabs from 'material-ui/lib/tabs/tabs';import Tab from 'material-ui/lib/tabs/tab';import Slider from 'material-ui/lib/slider';import Checkbox from 'material-ui/lib/checkbox';import DatePicker from 'material-ui/lib/date-picker/date-picker';import injectTapEventPlugin from 'react-tap-event-plugin';injectTapEventPlugin();const styles = { headline: { fontSize: 24, paddingTop: 16, marginBottom: 12, fontWeight: 400 }};const TabsSimple = React.createClass({ render: () => ( <Tabs> <Tab label="Item One"> <div> <h2 style={styles.headline}>Tab One Template Example</h2> <p> This is the first tab. </p> <p> This is to demonstrate how easy it is to build mobile apps with react </p> <Slider name="slider0" defaultValue={0.5}/> </div> </Tab> <Tab label="Item 2"> <div> <h2 style={styles.headline}>Tab Two Template Example</h2> <p> This is the second tab </p> <Checkbox name="checkboxName1" value="checkboxValue1" label="Installed Cordova"/> <Checkbox name="checkboxName2" value="checkboxValue2" label="Installed React"/> <Checkbox name="checkboxName3" value="checkboxValue3" label="Built the app"/> </div> </Tab> <Tab label="Item 3"> <div> <h2 style={styles.headline}>Tab Three Template Example</h2> <p> Choose a Date:</p> <DatePicker hintText="Select date"/> </div> </Tab> </Tabs> )});module.exports = TabsSimple; This file has quite a lot going on, so let’s break it down step by step: We import all the components that we're going to use in our app. This includes tabs, sliders, checkboxes, and datepickers. injectTapEventPlugin is a plugin that we need in order to get tab switching to work. We decide the style used for our tabs. Next, we make our Tabs react component, which consists of three tabs: The first tab has some text along with a slider. The second tab has a group of checkboxes. The third tab has a pop-up datepicker. Each component has a few keys, which are specific to it (such as the initial value of the slider, the value reference of the checkbox, or the placeholder for the datepicker). There are a lot more properties you can assign, which are specific to each component. Building your App For building on Android, you will first need to install the Android SDK. Now that we have all the code in place, all that is left is building the app. For this, make a new directory, start a new cordova project, and add the Android platform, by running the following on your terminal: mkdir my-cordova-project cd my-cordova-project cordova create . cordova platform add android Once the installation is complete, build the code we just wrote previously. If you are using the same build system as the source code, you will have only two files, that is, index.html and bundle.min.js. Delete all the files that are currently present in the www folder of your cordova project and copy those two files there instead. You can check whether your app is working on your computer by running cordova serve and going to the appropriate address on your browser. If all is well, you can build and deploy your app: cordova build android cordova run android This will build and install the app on your Android device (provided it is in debug mode and connected to your computer). Similarly, you can build and install the same app for iOS or windows (you may need additional tools such as XCode or .NET for iOS or Windows). You can also use any other framework to build your mobile app. The angular framework also comes with its own set of material design components. About the Author Soham Kamani is a full-stack web developer and electronics hobbyist.  He is especially interested in JavaScript, Python, and IoT.
Read more
  • 0
  • 0
  • 14657

article-image-network-virtualization-and-vsphere
Packt
21 Oct 2013
12 min read
Save for later

Network Virtualization and vSphere

Packt
21 Oct 2013
12 min read
(For more resources related to this topic, see here.) Network virtualization is what makes vCloud Director such an awesome tool. When we talk about isolated networks, we are talking about vCloud Director making use of different methods of the Network layer 3 encapsulation (OSI/ISO model). Basically, it's the same concept that was introduced with VLANs. VLANs split up the network communication in a network in different totally-isolated communication streams. vCloud makes use of these isolated networks to create networks in Organizations and vApps. vCloud Director has three different network items listed as follows: External Network: This is a network that exists outside vCloud, for example, a production network. It is basically a port group in vSphere that is used in vCloud to connect to the outside world. An External Network can be connected to multiple Organization Networks. External Networks are not virtualized and are based on existing port groups on vSwitch or a Distributed Switch (also called a vNetwork Distributed Switch or vNDS). Organization Network: This is a network that exists only inside one organization. You can have multiple Organization Networks in an organization. Organizational networks come in three different types: Isolated: An isolated Organization Network exists only in this organization and is not connected to an External Network; however, it can be connected to vApp Networks or VMs. This network type uses network virtualization and its own network settings. Routed Network (Edge Gateway): An Organization Network connects to an existing Edge Device. An Edge Gateway allows defining firewall, NAT rules, DHCP services, Static Routes, as well as VPN connections and the load balance functionality. Routed Gateways connect External Networks to vApp Networks and/or VMs. This network uses virtualized networks and its own network settings. Directly connected: This Organization Network is an extension of an External Network into the organization. They directly connect External Networks to vApp Networks or VMs. These networks do NOT use network virtualization and they make use of the network settings of an External Network. vApp Network: This is a virtualized network that only exists inside a vApp. You can have multiple vApp Networks inside one vApp. A vApp Network can connect to VMs and to Organization Networks. It has its own network settings. When connecting a vApp Network to an Organization Network, you can create a router between the vApp and the Organization Network, which lets you define DHCP, firewall, NAT rules, and Static Routing. To create isolated networks, vCloud Director uses Network Pools. Network Pools are a collection of VLANs, port groups, and VLANs that can use layer 2 in the layer 3 encapsulation. The content of these pools can be used by Organizations and vApp Networks for network virtualization. Network Pools There are four kinds of Network Pools that can be created: Virtual eXtensible LANs (VXLAN): VXLAN networks are layer 2 networks that are encapsulated in layer 3 packets. VMware calls this Software Defined Networking (SDN). VXLANs are automatically created by vCloud Director (vCD); however, they don't work out of the box and require some extra configuration in vCloud Network and Security (refer to the Making VXLANs work recipe). Network isolation-backed: These have basically the same concept as VXLANs; however, they work out of the box and use MAC-in-MAC encapsulation. The difference is that VXLANs can transcend routers whereas Network isolation-backed networks can't (refer to the Creating isolated networks without 1,000 VXLANs recipe). vSphere port groups-backed: vCD uses pre-created port groups to build the vApp or Organization Networks. You need to pre-provision one port group for every vApp/Organization Network you would like to use. VLAN-backed: vCD uses a pool of VLAN numbers to automatically provision port groups on demand; however, you still need to configure the VLAN trunking. You will need to reserve one VLAN for every vApp/Organization Network you would like to use. VXLANs and Network isolation-backed networks solve the problems of pre-provisioning and reserving a multitude of VLANs, which makes them extremely important. However, using a port group or VLAN Network Pools can have additional benefits that we will explore later. So let's get started! Now let's have a closer look at what one can do with networks in vCloud, but before we dive into the recipes, let's make sure we are all on the same page. Usage of different Network types vCloud Director has three different network items. An External Network is basically a port group in vSphere that is imported into vCloud. An Organization Network is an isolated network that exists only in an organization. The same is true for vApp Networks, which exists only in vApps. In each example you will also see a diagram of the specific network: Isolated vApp Network Isolated vApp Networks exist only inside vApps. They are useful if one needs to test how VMs behave in a network or to test using an IP range that is already in use (for example, production). The downside of them is that they are isolated, meaning that it is hard to get information or software in and out. Have a look at the Forwarding an RDP (or SSH) session into an isolated vApp and accessing a fully isolated vApp or Organization Network recipes in this article to find some answers to this problem. VMs directly connected to an External Network VMs inside a vApp are connected to a Direct Organization Network that is again directly connected to an External Network, meaning that they will use the IPs from the External Network Pool. Typically, these VMs are used for production, making it possible for customers to choose vCloud for fast provisioning of preconfigured templates. As vCloud manages the IPs for a given IP range (Static Pool), it can be quite easy to fast provision multiple VMs this way. vApp Network connected via vApp router to an External Network VMs are connected to a vApp Network that has a vApp router defined as its gateway. The gateway connects to a Direct Organization Network. The gateway will automatically be given an IP from the External Network Pool. The IPs of the VMs inside the vApp will be managed by the vApp Static Pool. These configurations come in handy to reduce the amount of physical networking that has to be provisioned. The vApp router can act as a router with defined firewall rules, it can do S-NAT and D-NAT as well as define static routing and DHCP services. So instead of using a physical VLAN or subnet, one can hide away applications this way. As an added benefit, these applications can be used as templates for fast deployment. VMs directly connected to an isolated Organization Network VMs are connected directly to an isolated Organization Network. Connecting VMs directly to an isolated Organization Network normally only makes sense if there's more than one vApp/VM connected to the same Organization Network. These network constructs come in handy when we want to repeatedly test complex applications that require certain infrastructure services such as Active Directory, DHCP, DNS, database, and Exchange Servers. Instead of deploying the needed infrastructure inside the testing vApp, we create a new vApp that contains only the infrastructure. By connecting the test vApp to the infrastructure vApp via an isolated Organization Network, the test vApp can now use the infrastructure. This makes it possible to re-use these infrastructure services not only for one vApp but also for many vApps, reducing the amount of resources needed for testing. By using vApp sharing options, you can even hide away the infrastructure vApp from your users. vApp connected via a vApp router to an isolated Organization Network VMs are connected to a vApp Network that has a vApp router as its gateway. The vApp router gets its IP automatically from the Organization Network pool. The VMs will get their IPs from the vApp Network pool. Basically, it is a combination of the network examples—VMs directly connected to an isolated Organization Network and a vApp Network connected via a vApp router to an External Network. A test vApp or an infrastructure vApp can be packaged this way and be made ready for fast deployment. VMs connected directly to an Edge device. VMs are directly connected to the Edge Organization Network and get their IPs from the Organization Network pool. Their gateway is the Edge device that connects them to the External Networks through the Edge firewall. A typical example for this is the usage of the Edge load balancing feature in order to load balance VMs inside the vApp. Another example is that organizations that are using the same External Network are secured against each other using the Edge firewall. This is mostly the case if the External Network is the Internet and each organization is an external customer. A vApp connected to an Edge via a vApp router. VMs are connected to a vApp Network that has the vApp router as its gateway. The vApp router will automatically get an IP from the Organization Network, which again has its gateway as the Edge. The VMs will get their IPs from the vApp Network pool. This is a more complicated variant of the previous example, allowing customers to package their VMs, secure them against other vApps or VMs, or subdivide their allocated networks. IP management Let's have a look at IP management with vCloud. vCloud has the following three different settings for IP management of VMs: DHCP: You will need to provide a DHCP as vCloud doesn't automatically create one. However, a vApp router or an Edge can create one. Static-IP Pool: The IP for the VM comes from the Static IP Pool of the network it is connected to. In addition to the IP, the subnet mask, DNS, gateway, and domain suffix will be configured on the VM according to the IP settings. Static-Manual: The IP can be defined manually; it doesn't come from the pool. The IP you define must be part of the network segment that is defined by the gateway and the subnet mask. In addition to the IP, the subnet mask, DNS, gateway, and domain suffix will be configured on the VM according to the IP settings. All these settings require Guest Customization to be effective. If no Guest Customization is selected, or if the VM doesn't have VMware tools installed, it doesn't work, and whatever the VM was configured with as a template will be used. Instead of wasting space and retyping what you need for each recipe every time, the following are some of the basic ingredients you will have to have ready for this article. An organization in which at least one OvDC is present. The OvDC needs to be configured with at least three free isolated networks that have a network pool defined. Some VM templates of an OS type you find easy to use (Linux or Windows) An External Network that connects you to the outside world (as in outside vCloud), for example, your desktop, and has at least five IPs in the Static IP Pool. One thing that needs to be said about vApps is that they actually come in two completely different versions: the vSphere vApp and the vCloud vApp. vSphere and vCloud vApps The vSphere vApp concept was introduced in vSphere 4.0 as a container for VMs. In vSphere, a vApp is essentially a resource pool with some extras, such as the starting and stopping order and (if you configured it) network IP allocation methods. The idea is for the vApp to be an entity of VMs that build one unit. Such vApps can then be exported or imported using OVF (Open Virtualization Format). A very good example of a vApp is VMware Operations Manager. It comes as a vApp in an OVF and contains not only the VMs but also the startup sequence as well as setup scripts. When the vApp is deployed for the first time, additional information such as network settings are asked and then implemented. A vSphere vApp is a resource pool; it can be configured so that it will only demand resources that it is using; on the other hand, resource pool configuration is something that most people struggle with. A vSphere vApp is only a resource pool; it is not automatically represented as a folder within the VMs and Template view of vSphere, but is viewed there as a vApp, as shown in the following screenshot: The vCloud vApp is a very different concept. First of all, it is not a resource pool. The VMs of the vCloud vApp live in the OvDC resource pool. However, the vCloud vApp is automatically a folder in the VMs and Template view of vSphere. It is a construct that is created by vCloud, and consists of VMs, a start and stop sequence, and networks. The network part is one of the major differences (next to the resource pool). In vSphere, only basic network information (IP's assignment, gateway, and DNS) is stored in the vApp. A vCloud vApp actually encapsulates the networks. The vCloud vApp networks are full networks, meaning they contain the full information for a given network including network settings and IP pools. This information is kept while importing and exporting vCloud vApps, as shown in the following screenshot: While I'm referring to vApps in this article, I will always mean vCloud vApps. If vCenter vApps feature anywhere in this article, they will be written as vCenter vApp. Summary In this article we learned different VMware concepts that will help in improving productivity. We also went through recipes that deal with the daily tasks and also present new ideas and concepts that you may not have thought of before. Resources for Article: Further resources on this subject: Windows 8 with VMware View [Article] Cloning and Snapshots in VMware Workstation [Article] vCloud Networks [Article]
Read more
  • 0
  • 0
  • 14654

article-image-designing-games-swift
Packt
03 Nov 2016
16 min read
Save for later

Designing Games with Swift

Packt
03 Nov 2016
16 min read
In this article by Stephen Haney, the author of the book Swift 3 Game Development - Second Edition, we will see that apple's newest version of its flagship programming language, Swift 3, is the perfect choice for game developers. As it matures, Swift is realizing its opportunity to be something special, a revolutionary tool for app creators. Swift is the gateway for developers to create the next big game in the Apple ecosystem. We have only started to explore the wonderful potential of mobile gaming, and Swift is the modernization we need for our toolset. Swift is fast, safe, current, and attractive to developers coming from other languages. Whether you are new to the Apple world, or a seasoned veteran of Objective-C, I think you will enjoy making games with Swift. (For more resources related to this topic, see here.) Apple's website states the following: "Swift is a successor to the C and Objective-C languages." My goal is to guide you step-by-step through the creation of a 2D game for iPhones and iPads. We will start with installing the necessary software, work through each layer of game development, and ultimately publish our new game to the App Store. We will also have some fun along the way! We aim to create an endless flyer game featuring a magnificent flying penguin named Pierre. What is an endless flyer? Picture hit games like iCopter, Flappy Bird, Whale Trail, Jetpack Joyride, and many more—the list is quite long. Endless flyer games are popular on the App Store, and the genre necessitates that we cover many reusable components of 2D game design. I will show you how to modify our mechanics to create many different game styles. My hope is that our demo project will serve as a template for your own creative works. Before you know it, you will be publishing your own game ideas using the techniques we explore together. In this article, we will learn the following topics: Why you will love Swift What you will learn in this article New in Swift 3 Setting up your development environment Creating your first Swift game Why you will love Swift Swift, as a modern programming language, benefits from the collective experience of the programming community; it combines the best parts of other languages and avoids poor design decisions. Here are a few of my favorite Swift features: Beautiful syntax: Swift's syntax is modern and approachable, regardless of your existing programming experience. Apple balanced syntax with structure to make Swift concise and readable. Interoperability: Swift can plug directly into your existing projects and run side-by-side with your Objective-C code. Strong typing: Swift is a strongly typed language. This means the compiler will catch more bugs at compile time, instead of when your users are playing your game! The compiler will expect your variables to be of a certain type (int, string, and so on) and will throw a compile-time error if you try to assign a value of a different type. While this may seem rigid if you are coming from a weakly typed language, the added structure results in safer, more reliable code. Smart type inference: To make things easier, type inference will automatically detect the types of your variables and constants based upon their initial value. You do not need to explicitly declare a type for your variables. Swift is smart enough to infer variable types in most expressions. Automatic memory management: As the Apple Swift developer guide states, "memory management just works in Swift". Swift uses a method called Automatic Reference Counting (ARC) to manage your game's memory usage. Besides a few edge cases, you can rely on Swift to safely clean up and turn off the lights. An even playing field: One of my favorite things about Swift is how quickly the language is gaining mainstream adoption. We are all learning and growing together, and there is a tremendous opportunity to break new ground. Open source: From version 2.2 onwards, Apple made Swift open source, curetting it through the website www.swift.org, and launched a package manager with Swift 3. This is a welcome change as it fosters greater community involvement and a larger ecosystem of third party tools and add-ons. Eventually, we should see Swift migrate to new platforms. Prerequisites I will try to make this text easy to understand for all skill levels: I will assume you are brand new to Swift as a language Requires no prior game development experience, though it will help I will assume you have a fundamental understanding of common programming concepts What you will learn in this article You will be capable of creating and publishing your own iOS games. You will know how to combine the techniques we learned to create your own style of game, and you will be well prepared to dive into more advanced topics with a solid foundation in 2D game design. Embracing SpriteKit SpriteKit is Apple's 2D game development framework and your main tool for iOS game design. SpriteKit will handle the mechanics of our graphics rendering, physics, and sound playback. As far as game development frameworks go, SpriteKit is a terrific choice. It is built and supported by Apple and thus integrates perfectly with Xcode and iOS. You will learn to be highly proficient with SpriteKit as we will be using it exclusively in our demo game. We will learn to use SpriteKit to power the mechanics of our game in the following ways: Animate our player, enemies, and power-ups Paint and move side-scrolling environments Play sounds and music Apply physics-like gravity and impulses for movement Handle collisions between game objects Reacting to player input The control schemes in mobile games must be inventive. Mobile hardware forces us to simulate traditional controller inputs, such as directional pads and multiple buttons, on the screen. This takes up valuable visible area, and provides less precision and feedback than with physical devices. Many games operate with only a single input method: A single tap anywhere on the screen. We will learn how to make the best of mobile input, and explore new forms of control by sensing device motion and tilt. Structuring your game code It is important to write well-structured code that is easy to re-use and modify as your game design inevitably changes. You will often find mechanical improvements as you develop and test your games, and you will thank yourself for a clean working environment. Though there are many ways to approach this topic, we will explore some best practices to build an organized system with classes, protocols, inheritance, and composition. Building UI/menus/levels We will learn to switch between scenes in our game with a menu screen. We will cover the basics of user experience design and menu layout as we build our demo game. Integrating with Game Center Game Center is Apple's built-in social gaming network. Your game can tie into Game Center to store and share high scores and achievements. We will learn how to register for Game Center, tie it into our code, and create a fun achievement system. Maximizing fun If you are like me, you will have dozens of ideas for games floating around your head. Ideas come easily, but designing fun game play is difficult! It is common to find that your ideas need game play enhancements once you see your design in action. We will look at how to avoid dead-ends and see your project through to the finish line. Plus, I will share my tips and tricks to ensure your game will bring joy to your players. Crossing the finish line Creating a game is an experience you will treasure. Sharing your hard work will only sweeten the satisfaction. Once our game is polished and ready for public consumption, we will navigate the App Store submission process together. You will finish feeling confident in your ability to create games with Swift and bring them to market in the App Store. Monetizing your work Game development is a fun and rewarding process, even without compensation, but the potential exists to start a career, or side-job, selling games on the App Store. Successfully promoting and marketing your game is an important task. I will outline your options and start you down the path to monetization. New in Swift 3 The largest feature in Swift 3 is syntax compatibility and stability. Apple is trying to refine its young, shifting language into its final foundational shape. Each successive update of Swift has introduced breaking syntax changes that made older code incompatible with the newest version of Swift; this is very inconvenient for developers. Going forward, Swift 3 aims to reach maturity and maintain source compatibility with future releases of the language. Swift 3 also features the following:  A package manager that will help grow the ecosystem A more consistent, readable API that often results in less code for the same result Improved tooling and bug fixes in the IDE, Xcode Many small syntax improvements in consistency and clarity Swift has already made tremendous steps forward as a powerful, young language. Now Apple is working on polishing Swift into a mature, production-ready tool. The overall developer experience improves with Swift 3. Setting up your development environment Learning a new development environment can be a roadblock. Luckily, Apple provides some excellent tools for iOS developers. We will start our journey by installing Xcode. Introducing and installing Xcode Xcode is Apple's Integrated Development Environment (IDE). You will need Xcode to create your game projects, write and debug your code, and build your project for the App Store. Xcode also comes bundled with an iOS simulator to test your game on virtualized iPhones and iPads on your computer. Apple praises Xcode as "an incredibly productive environment for building amazing apps for Mac, iPhone, and iPad".   To install Xcode, search for Xcode in the AppStore, or visit http://developer.apple.com and select Developer and then Xcode. Swift is continually evolving, and each new Xcode release brings changes to Swift. If you run into errors because Swift has changed, you can always use Xcode's built-in syntax update tool. Simply use Xcode's Edit | Convert to Latest Syntax option to update your code. Xcode performs common IDE features to help you write better, faster code. If you have used IDEs in the past, then you are probably familiar with auto completion, live error highlighting, running and debugging a project, and using a project manager pane to create and organize your files. However, any new program can seem overwhelming at first. We will walk through some common interface functions over the next few pages. I have also found tutorial videos on YouTube to be particularly helpful if you are stuck. Most common search queries result in helpful videos. Creating our first Swift game Do you have Xcode installed? Let us see some game code in action in the simulator! We will start by creating a new project in Xcode. For our demo game, we will create a side-scrolling endless flyer featuring an astonishing flying penguin named Pierre. I am going to name this project Pierre Penguin Escapes the Antarctic, but feel free to name your project whatever you like. Follow these steps to create a new project in Xcode: Launch Xcode and navigate to File | New | Project. You will see a screen asking you to select a template for your new project. Select iOS | Application in the left pane, and Game in the right pane. It should look like this: Once you select Game, click Next. The following screen asks us to enter some basic information about our project. Don’t worry; we are almost at the fun bit. Fill in the Product Name field with the name of your game. Let us fill in the Team field. Do you have an active Apple developer account? If not, you can skip over the Team field for now. If you do, your Team is your developer account. Click Add Team and Xcode will open the accounts screen where you can log in. Enter your developer credentials as shown in the following screenshot: Once you're authenticated, you can close the accounts screen. Your developer account should appear in the Team dropdown. You will want to pick a meaningful Organization Name and Organization Identifier when you create your own games for publication. Your Organization Name is the name of your game development studio. For me, that's Joyful Games. By convention, your Organization Identifier should follow a reverse domain name style. I will use io.JoyfulGames since my website is JoyfulGames.io. After you fill out the name fields, be sure to select Swift for the Language, SpriteKit for Game Technology, and Universal for Devices. For now, uncheck Integrate GameplayKit, uncheck Include Unit Tests, uncheck Include UI Tests. We will not use these features in our demo game. Here are my final project settings: Click Next and you will see the final dialog box. Save your new project. Pick a location on your computer and click Next. And we are in! Xcode has pre-populated our project with a basic SpriteKit template. Navigating our project Now that we have created our project, you will see the project navigator on the left-hand side of Xcode. You will use the project navigator to add, remove, and rename files and generally organize your project. You might notice that Xcode has created quite a few files in our new project. We will take it slow; don’t feel that you have to know what each file does yet, but feel free to explore them if you are curious: Exploring the SpriteKit Demo Use the project navigator to open up the file named GameScene.swift. Xcode created GameScene.swift to store the default scene of our new game. What is a scene? SpriteKit uses the concept of scenes to encapsulate each unique area of a game. Think of the scenes in a movie; we will create a scene for the main menu, a scene for the Game Over screen, a scene for each level in our game, and so on. If you are on the main menu of a game and you tap Play, you move from the menu scene to the Level 1 scene. SpriteKit prepends its class names with the letters "SK"; consequently, the scene class is SKScene. You will see there is already some code in this scene. The SpriteKit project template comes with a very small demo. Let's take a quick look at this demo code and use it to test the iOS simulator. Please do not be concerned with understanding the demo code at this point. Your focus should be on learning the development environment. Look for the run toolbar at the top of the Xcode window. It should look something like the following: Select the iOS device of your choice to simulate using the dropdown on the far right. Which iOS device should you simulate? You are free to use the device of your choice. I will be using an iPhone 6 for the screenshots, so choose iPhone 6 if you want your results to match my images perfectly. Unfortunately, expect your game to play poorly in the simulator. SpriteKit suffers from poor FPS in the iOS simulator. Once our game becomes relatively complex, we will see our FPS drop, even on high-end computers. The simulator will get you through, but it is best if you can plug in a physical device to test. It is time for our first glimpse of SpriteKit in action! Press the gray play arrow in the toolbar (handy keyboard shortcut: command + r). Xcode will build the project and launch the simulator. The simulator starts in a new window, so make sure you bring it to the front. You should see a gray background with white text: Hello, World. Click around on the gray background. You will see colorful, spinning boxes spawning wherever you click: If you have made it this far, congratulations! You have successfully installed and configured everything you need to make your first Swift game. Once you have finished playing with the spinning squares, you can close the simulator down and return to Xcode. Note: You can use the keyboard command command + q to exit the simulator or press the stop button inside Xcode. If you use the stop button, the simulator will remain open and launch your next build faster. Examining the demo code Let's quickly explore the demo code. Do not worry about understanding everything just yet; we will cover each element in depth later. At this point, I am hoping you will acclimatize to the development environment and pick up a few things along the way. If you are stuck, keep going! Make sure you have GameScene.swift open in Xcode. The demo GameScene class implements some functions you will use in your games. Let’s examine these functions. Feel free to read the code inside each function, but I do not expect you to understand the specific code just yet. The game invokes the didMove function whenever it switches to the GameScene. You can think of it a bit like an initialize, or main, function for the scene. The SpriteKit demo uses it to draw the Hello, World text to the screen and set up the spinning square shape that shows up when we tap. There are seven functions involving touch which handle the user's touch input to the iOS device screen. The SpriteKit demo uses these functions to spawn the spinning square wherever we touch the screen. Do not worry about understanding these functions at this time. The update function runs once for every frame drawn to the screen. The SpriteKit demo does not use this function, but we may have reason to implement it later. Cleaning up I hope that you have absorbed some Swift syntax and gained an overview of Swift and SpriteKit. It is time to make room for our own game; let us clear all of that demo code out! We want to keep a little bit of the boilerplate, but we can delete most of what is inside the functions. To be clear, I do not expect you to understand this code yet. This is simply a necessary step towards the start of our journey. Please remove lines from your GameScene.swift file until it looks like the following code: import SpriteKit class GameScene: SKScene { override funcdidMove(to view: SKView) { } } Summary You have already accomplished a lot. You have had your first experience with Swift, installed and configured your development environment, launched code successfully into the iOS simulator. Great work! Resources for Article: Further resources on this subject: Swift for Open Source Developers [Article] Swift Power and Performance [Article] Introducing the Swift Programming Language [Article]
Read more
  • 0
  • 0
  • 14638

article-image-introducing-iot-particles-photon-and-electron
Packt
07 Sep 2016
14 min read
Save for later

Introducing IoT with Particle's Photon and Electron

Packt
07 Sep 2016
14 min read
In this article by Rashid Khan, Kajari Ghoshdastidar, and Ajith Vasudevan, authors of the book Learning IoT with Particle Photon and Electron, we will have a brief walkthrough of the evolution of Internet of Things (IoT) followed by an overview of the basics of IoT-related software and hardware, which every IoT enthusiast should know. The discussion then moves on to introduce Particle, an IoT company (https://www.particle.io/), followed by a description of Particle's popular IoT products—Core, Photon and Electron. This article will cover following topics: Evolution of IoT Hardware and software in the IoT ecosystem Market survey of IoT development boards and cloud services What is Particle? Summary (For more resources related to this topic, see here.) Evolution of IoT It is not very clear exactly who coined the term IoT. Kevin Ashton (https://en.wikipedia.org/wiki/Kevin_Ashton) supposedly coined the phrase IoT while working for Procter & Gamble (P&G) in 1999. Kevin was then working on an RFID (https://en.wikipedia.org/wiki/Radio-frequency_identification) initiative by P&G, and proposed taking the system online to the Internet. In 2005, UN's International Telecommunications Union (ITU) - http://www.itu.int/, published its first report on IoT. In 2008, the global non-profit organization IPSO Alliance (http://www.ipso-alliance.org/) was launched to serve the various communities seeking to establish IoT by providing coordinated marketing efforts available to the general public. IPSO currently has more than 50 member companies including Google, Cisco, Intel, Texas Instruments, Bosch, Atmel. In 2012, IoT Consortium (IoTC) - http://iofthings.org/, was founded to educate technology firms, retailers, insurance companies, marketers, media companies, and the wider business community about the value of IoT. IoTC has more than 60 member companies in the area of hardware, software, and analytics, a few of them being Logitech, Node, and SigFox. A 2014 Forbes article by Gil Press mentions: "Gartner estimates that IoT product and service suppliers will generate incremental revenue exceeding $300 billion in 2020. IDC forecasts that the worldwide market for IoT solutions will grow from $1.9 trillion in 2013 to $7.1 trillion in 2020". Why IoT has become a household word now IoT has, in recent years, become quite popular and an everyday phenomenon primarily due to IoT-related hardware, software, accessories, sensors, and the Internet connection becoming very affordable and user friendly. An explosion in the availability of free Integrated Development Environments (IDEs) and Software Development Kits (SDKs) have made programming and deployment of IoT really simple and easy. Thus, IoT enthusiasts range from school kids, hobbyists, and non-programmers to embedded software engineers specialized in this area. Hardware and software in the IoT ecosystem Advancement in technology and affordability has made acquisition and usage of IoT devices very simple. However, in order to decide which IoT package (boards, accessories, sensors, software) to choose for a particular application, and actually building projects, it is essential to have knowledge of IoT terminology, hardware, and software. In this section, we will introduce the reader to the essential terminology used when dealing with IoT. This will also help the reader understand and appreciate the features of the Particle IoT products—Core, Photon, and Electron. Essential terminology Let's learn about a few terms that we're going to be hearing all throughout this article, and whenever we work with IoT hardware and software components: Term Definition IoT Development Board A development board is essentially a programmable circuit board which wraps an IoT device. The IoT device's processor/microcontroller, memory, communications ports, input-output pins, sensors, Wi-Fi module, and so on are exposed by the development board, in a convenient way, to the user. A board manufacturer usually provides an IDE with it to write and deploy code to the physical board. A development board with the IDE enables rapid prototyping of IoT projects. Microcontroller A microcontroller is a highly compact single Integrated Circuit (IC) with a processor and limited Random Access Memory (RAM) and Read Only Memory (ROM) embedded in it with programmable peripherals. Microcontrollers are "computer on a single chip". Because of its limited memory and architecture constraints, usually, only one specific program is deployable and runnable on a microcontroller at one time. Preprogrammed microcontrollers are used in electrical machinery such as washing machines, dish-washers, microwave, and so on. Microprocessor A microprocessor is a single integrated chip which in itself is a Central Processing Unit (CPU). The microprocessor has separate RAM and ROM modules, and digital inputs and outputs. The Microprocessor CPU is usually more powerful than that of a microcontroller, and there is provision to add larger amounts of memory externally. This makes microprocessors suitable for general-purpose programming, and are used in desktop computers, Laptops, and the like. Flash Memory Flash memory is an electronic non-volatile storage device, for example, USB pen-drives, memory cards, and so on. Data in Flash memory can be erased and rewritten. Unlike RAM, access speed is lower for flash memories, and also unlike RAM, the data stored in flash memory is not erased when power is switched off. Flash memories are generally used as reusable extra storage. RTOS RTOS as the name suggests, RTOS responds to events in real time. This means, as soon as an event occurs, a response is guaranteed within an acceptable and calculable amount of time. RTOS can be hard, firm, or soft depending on the amount of flexibility allowed in missing a task deadline. RTOS is essential in embedded systems, where real-time response is necessary. M2M Machine-to-Machine (M2M) communication encompasses communication between two or more machines (devices, computers, sensors, and so on) over a network (wireless/wired). Basically, a variant of IoT, where things are machines. Cloud Technology Cloud refers to computing resources available for use over a network (usually, the Internet). An end user can use such a resource on demand without having to install anything more than a lightweight client in the local machine. The major resources relevant to IoT include data storage, data analytics, data streaming, and communication with other devices. mBaaS Mobile Backend as a Service (mBaaS) is a infrastructure that provides cloud storage, data streaming, push notifications, and other related services for mobile application developers (web, native, IoT app development). The services are exposed via web-based APIs. BaaS is usually provided as a pay-per-use service. GPIO General Purpose Input Output (GPIO), these are electrical terminals or 'pins' exposed from ICs and IoT devices/boards that can be used to either send a signal to the device from the outside (input mode), or get a signal out from the inside of the device (output mode). Input or Output mode can be configured by the user at runtime. Module Unit of electronics, sometimes a single IC and at other times a group of components that may include ICs, providing a logical function to the device/board. For example, a Wi-Fi module provides Wi-Fi functionality to a board. Other examples are Bluetooth, Ethernet, USB, and so on, Port An electrical or Radio-Frequency-based interface available on a board through which external components can communicate with the board. For example, HDMI, USB, Ethernet, 3.5mm jack, UART (https://en.wikipedia.org/wiki/Universal_asynchronous_receiver/transmitter). Table 1: Terminology Network Protocols Connected smart devices need to communicate with each other, and exchange large volumes of messages between themselves and the cloud. To ensure near real-time response, smart bandwidth usage, and energy savings on the resource-constrained IoT devices, new protocols have been added to the traditional seven-layer network model (OSI model: https://en.wikipedia.org/wiki/OSI_model). The following table shows the major OSI network protocols and the IoT network protocols suitable for various smart, connected devices. Layer Examples of Traditional Network Protocols (OSI) Examples of IoT Network Protocols Application, Presentation, Session HTTP, FTP, SMTP, TLS, RPC, JSON, CSS, GIF, XML CoAP, MQTT, DDS, M2M service layer Transport TCP, UDP UDP, DTLS Network ICMP, IPsec, IPv4, IPv6 6LoWPAN, RPL (Zigbee) Data Link IEEE 802.2, L2TP, LLDP, MAC, PPP IEEE 802.15.4, BLE4.0, RFID, NFC, Cellular Physical DSL, Ethernet physical layer, RS-232, any physical transmission medium (for example, Cables) Wires, Sensor drivers to read from sensor devices Table 2: Layerwise Network Protocols – OSI vs IoT Market survey of IoT development boards and cloud services Here we list some of the most popular IoT boards and cloud services, available in the market at the time of writing this article, with some of their important specifications and features. These tables should help the reader get an idea as to where Particle products fit in on the IoT map. IoT development boards The next table lists the main specifications of popular IoT boards. These specs are the basic details one has to consider while selecting a board—its specifications in terms of processor and speed, memory, available communication modules and ports, and IO Pins. Also, while selecting a board, one has to analyze and match the project's requirements with the available boards, so that the right board is selected for the application in terms of fitment and performance. Board Name Microcontroller Microprocessor Memory Modules Ports IO Pins Raspberry Pi 1/2/3 Broadcom SoC BCM2835/6/7 Single/Quad-core ARM 11/Cortex-A7/A53 CPU, VideoCore IV GPU 256MB/512MB/1 GB RAM Ethernet, Wi-Fi, Serial UART, I2C HDMI, USB, Ethernet (RJ45), GPIO 26/40/40 Arduino Mini ATmega328 NA 32 KB Flash 2 KB SRAM NA NA 14 Arduino Yun ATmega32u4 Atheros AR9331 32 KB Flash 2.5 KB SRAM, 16 MB Flash, 64 MB RAM Wi-Fi, Ethernet USB, Ethernet (RJ45) 20 Intel Edison MCU at 100 MHz ( Intel Atom Soc) Dual-core CPU at 500 MHz (Intel Atom Soc) 4 GB Flash, 1 GB RAM Wi-Fi, Bluetooth 4.0 USB, UART, SPI, GPIO 28 Libelium Waspmote ATmega1281 NA 128 KB Flash, 8 KB SRAM Temp, Humidity, Light Sensors, (optional) GPS UART, I2C, SPI, USB 19 NodeMCU ESP8266 ESP 8266 SoC ESP-12 module 4 MB Flash Wi-Fi, Serial UART, ADC UART, GPIO, SPI 14 BeagleBone Black Sitara SoC AM3358/9 AM335x 1 GHz ARM Cortex-A8 512 MB RAM, 2/4 GB flash storage Ethernet, Serial UART, ADC, I2C Ethernet (RJ45), HDMI, USB, GPIO 24 CubieBoard ARM Cortex-A8 CPU AllWinner A10 SoC 512 MB/ 1 GB RAM, 4 GB flash memory Ethernet, Serial UART, ADC, I2C Ethernet (RJ45) , USB, SATA 96 Table 3: IoT development Boards Cloud services (PaaS, BaaS, M2M) It is important to know what kind of cloud service we will be dealing with, and whether our board has open standards and allows us to use our own personal service easily, or whether the board-provided service needs some manipulation to use in the current project. Cloud Service Name Salient Features Amazon Web Services (https://aws.amazon.com/) Microsoft Azure (https://azure.microsoft.com/) Cloud Foundry (https://www.cloudfoundry.org/) IBM Bluemix (http://www.ibm.com/cloud-computing/bluemix/) Platform as a Service (PaaS) Infrastructure (VM, Storage), Big Data Analytics, Application Services, Deployment and Management, Mobile and Device Services Parse (http://www.parse.com/) Kinvey (http://www.kinvey.com/) AnyPresence (http://www.anypresence.com/) Appcelerator (http://www.appcelerator.com/) mBaaS ThingWorx (https://www.thingworx.com/) M2M offering from PTC (http://www.ptc.com/) Table 4: Cloud services What is Particle? Particle (https://www.particle.io), formerly known as Spark, is a company started by Zach Supalla. It provides hardware and software for development of IoT projects. Journey of Particle The first company started by Zach Supalla in 2011 was known as Hex Goods, and it sold designer products online. In early 2012, Hex Goods was shut down, and Zach started a second company called Switch Devices, which dealt with connected lighting. Switch Devices was then renamed Spark Devices. The name Spark was used as it provided a double meaning to the founders. Spark stood for spark of light and also sparks of inspiration. In early 2013, Spark transformed to an IoT platform for engineers and developers. The name Spark also did not last long as the founders felt Spark created confusion for a lot of users. There exist 681 live trademarks that include the word Spark. Apart from the number of trademarks, there are some other great, unrelated software and hardware products employing the name Spark in them—some of them being Apache Spark, SparkFun, and Spark NZ. It has been reported that a lot of people logged on to Zach's #spark IRC channel and asked doubts about big data. The name Particle was finally chosen, as it gave plenty of room to grow in terms of products and offerings. Particle, in scientific terms, is a single discreet unit within a larger system. The name draws a parallel with the mission of Particle—the company which provides development kits and devices as single units used to build the greater whole of IoT. We'll cover Particle IoT products in depth, and see how and when they perform better than other IoT development boards. Why Particle? Today, the most recurring problem with all existing IoT prototyping boards is that of connectivity. In order to connect the existing boards to the Internet, additional components such as Wi-Fi or GSM modules have to be attached in the development environment as well as in production. Attaching external devices for communication is cumbersome, and adds another point of failure with frequent issues such as Internet unavailability, intermittent network connectivity, and so on. This leads to a bad experience for the developer. Developers have to frequently (re)write code, deploy it onto the device(s), test, debug, fix any bugs, rinse, and repeat. The problem with code deployment with existing boards is that the boards need to be connected to a computer, which means for even the smallest code update, the device/board needs to be connected to the developer's computer, either by moving the computer to the device (which may be located at a not-so-easily accessible location) or vice versa. This poses a problem when the device, after an update at the developer's site, has to be placed back in its original production environment for testing and debugging the new changes. This means large turnaround times to load new code into production. Particle provides products that have built-in Wi-Fi modules or GSM modules, which help in easy connection to a network or the internet, with support for OTA (Over-The-Air) code deployment. This removes the hassle of adding extra modules on the prototyping boards for connectivity, and it also allows for pushing code or testing/debugging on site. As previously mentioned, one of the important features which differentiates Particle products from other devices is the Particle device's ability of deployment of code over the air. New code can be deployed onto the device or burnt, as the process is called in embedded systems' parlance, via REST API calls, which makes it very convenient to provide updates. This feature of Particle products helps in faster code release cycle and testing/debugging. What does Particle offer? Particle offers a suite of hardware and software tools to help prototype, scale, and manage the IoT products. It also provides the ability to build cloud-connected IoT prototypes quickly. If you're satisfied with your prototype and want to productize your IoT design, no problem there. It helps us to go from a single prototype to millions of units with a cloud platform that can scale as the number of devices grow. The popular Particle hardware devices are listed as follows: Core: A tiny Wi-Fi development kit for prototyping and scaling your IoT product. Reprogrammable and connected to the cloud, this has now been superseded by the Photon. Photon: A tiny Wi-Fi development kit for prototyping and scaling your IoT product. Reprogrammable and connected to the cloud. Electron: A tiny development kit for creating 2G/3G cellular connected products. The Photon and the Core are bundled with Wi-Fi modules, which help them connect to a network or the Internet without adding any extra modules. The Electron has a 3G/2G GSM module, which can be used to send or receive messages directly or connect to the Internet. The firmware for the Photon, Electron, and Core can be written in a web-based IDE provided by Particle, and the deployment of the firmware code to the device is done over-the-air. Particle also offers SDKs for mobile and web to extend the IoT experience from the devices/sensors to the phone and web. Summary In this article, we learnt about IoT, and how it all began. We briefly touched upon major organizations involved in IoT, common terminology used, and we looked at different hardware products and cloud services we have available for building IoT projects.  Resources for Article: Further resources on this subject: Identity and Access-Management Solutions for the IoT [article] Internet of Things with BeagleBone [article] The Internet of Things [article]
Read more
  • 0
  • 0
  • 14637
article-image-classes-and-instances-ember-object-model
Packt
05 Feb 2016
12 min read
Save for later

Classes and Instances of Ember Object Model

Packt
05 Feb 2016
12 min read
In this article by Erik Hanchett, author of the book Ember.js cookbook, Ember.js is an open source JavaScript framework that will make you more productive. It uses common idioms and practices, making it simple to create amazing single-page applications. It also let's you create code in a modular way using the latest JavaScript features. Not only that, it also has a great set of APIs in order to get any task done. The Ember.js community is welcoming newcomers and is ready to help you when required. (For more resources related to this topic, see here.) Working with classes and instances Creating and extending classes is a major feature of the Ember object model. In this recipe, we'll take a look at how creating and extending objects works. How to do it Let's begin by creating a very simple Ember class using extend(), as follows: const Light = Ember.Object.extend({ isOn: false }); This defines a new Light class with a isOn property. Light inherits the properties and behavior from the Ember object such as initializers, mixins, and computed properties. Ember Twiddle Tip At some point of time, you might need to test out small snippets of the Ember code. An easy way to do this is to use a website called Ember Twiddle. From that website, you can create an Ember application and run it in the browser as if you were using Ember CLI. You can even save and share it. It has similar tools like JSFiddle; however, only for Ember. Check it out at http://ember-twiddle.com. Once you have defined a class you'll need to be able to create an instance of it. You can do this by using the create() method. We'll go ahead and create an instance of Light. constbulb = Light.create(); Accessing properties within the bulb instance We can access the properties of the bulb object using the set and get accessor methods. Let's go ahead and get the isOn property of the Light class, as follows: console.log(bulb.get('isOn')); The preceding code will get the isOn property from the bulb instance. To change the isOn property, we can use the set accessor method: bulb.set('isOn', true) The isOn property will now be set to true instead of false. Initializing the Ember object The init method is invoked whenever a new instance is created. This is a great place to put in any code that you may need for the new instance. In our example, we'll go ahead and add an alert message that displays the default setting for the isOn property: const Light = Ember.Object.extend({ init(){ alert('The isON property is defaulted to ' + this.get('isOn')); }, isOn: false }); As soon as the Light.create line of code is executed, the instance will be created and this message will pop up on the screen. The isON property is defaulted to false. Subclass Be aware that you can create subclasses of your objects in Ember. You can override methods and access the parent class by using the _super() keyword method. This is done by creating a new object that uses the Ember extend method on the parent class. Another important thing to realize is that if you're subclassing a framework class such as Ember.Component and you override the init method, you'll need to make sure that you call this._super(). If not, the component may not work properly. Reopening classes At anytime, you can reopen a class and define new properties or methods in it. For this, use the reopen method. In our previous example, we had an isON property. Let's reopen the same class and add a color property, as follows: To add the color property, we need to use the reopen() method: Light.reopen({ color: 'yellow' }); If required, you can add static methods or properties using reopenClass, as follows: Light.reopen({ wattage: 40 }); You can now access the static property: Light.wattage How it works In the preceding examples, we have created an Ember object using extend. This tells Ember to create a new Ember class. The extend method uses inheritance in the Ember.js framework. The Light object inherits all the methods and bindings of the Ember object. The create method also inherits from the Ember object class and returns a new instance of this class. The bulb object is the new instance of the Ember object that we created. There's more To use the previous examples, we can create our own module and have it imported to our project. To do this, create a new MyObject.js file in the app folder, as follows: // app/myObject.js import Ember from 'ember'; export default function() { const Light = Ember.Object.extend({ init(){ alert('The isON property is defaulted to ' + this.get('isOn')); }, isOn: false }); Light.reopen({ color: 'yellow' }); Light.reopenClass({ wattage: 80 }); const bulb = Light.create(); console.log(bulb.get('color')); console.log(Light.wattage); } This is the module that we can now import to any file of our Ember application. In the app folder, edit the app.js file. You'll need to add the following line at the top of the file: // app/app.js import myObject from './myObject'; At the bottom, before the export, add the following line: myObject(); This will execute the myObject function that we created in the myObject.js file. After running the Ember server, you'll see the isOn property defaulted to the false pop-up message. Working with computed properties In this recipe, we'll take a look at the computed properties and how they can be used to display data, even if that data changes as the application is running. How to do it Let's create a new Ember.Object and add a computed property to it, as shown in the following: Begin by creating a new description computed property. This property will reflect the status of isOn and color properties: const Light = Ember.Object.extend({ isOn: false, color: 'yellow', description: Ember.computed('isOn','color',function() { return 'The ' + this.get('color') + ' light is set to ' + this.get('isOn'); }) }); We can now create a new Light object and get the computed property description: const bulb = Light.create(); bulb.get('description'); //The yellow light is set to false The preceding example creates a computed property that depends on the isOn and color properties. When the description function is called, it returns a string describing the state of the light. Computed properties will observe changes and dynamically update whenever they occur. To see this in action, we can change the preceding example and set the isOn property to false. Use the following code to accomplish this: bulb.set('isOn', true); bulb.get('description') //The yellow light is set to true The description has been automatically updated and will now display that the yellow light is set to true. Chaining the Light object Ember provides a nice feature that allows computed properties to be present in other computed properties. In the previous example, we created a description property that outputted some basic information about the Light object, as follows: Let's add another property that gives a full description: const Light = Ember.Object.extend({ isOn: false, color: 'yellow', age: null, description: Ember.computed('isOn','color',function() { return 'The ' + this.get('color') + ' light is set to ' + this.get('isOn'); }), fullDescription: Ember.computed('description','age',function() { return this.get('description') + ' and the age is ' + this.get('age') }), }); The fullDescription function returns a string that concatenates the output from description with a new string that displays the age: const bulb = Light.create({age:22}); bulb.get('fullDescription'); //The yellow light is set to false and the age is 22 In this example, during instantiation of the Light object, we set the age to 22. We can overwrite any property if required. Alias The Ember.computed.alias method allows us to create a property that is an alias for another property or object. Any call to get or set will behave as if the changes were made to the original property, as shown in the following: const Light = Ember.Object.extend({ isOn: false, color: 'yellow', age: null, description: Ember.computed('isOn','color',function() { return 'The ' + this.get('color') + ' light is set to ' + this.get('isOn'); }), fullDescription: Ember.computed('description','age',function() { return this.get('description') + ' and the age is ' + this.get('age') }), aliasDescription: Ember.computed.alias('fullDescription') }); const bulb = Light.create({age: 22}); bulb.get('aliasDescription'); //The yellow light is set to false and the age is 22. The aliasDescription will display the same text as fullDescription since it's just an alias of this object. If we made any changes later to any properties in the Light object, the alias would also observe these changes and be computed properly. How it works Computed properties are built on top of the observer pattern. Whenever an observation shows a state change, it recomputes the output. If no changes occur, then the result is cached. In other words, the computed properties are functions that get updated whenever any of their dependent value changes. You can use it in the same way that you would use a static property. They are common and useful throughout Ember and it's codebase. Keep in mind that a computed property will only update if it is in a template or function that is being used. If the function or template is not being called, then nothing will occur. This will help with performance. Working with Ember observers in Ember.js Observers are fundamental to the Ember object model. In the next recipe, we'll take our light example and add in an observer and see how it operates. How to do it To begin, we'll add a new isOnChanged observer. This will only trigger when the isOn property changes: const Light = Ember.Object.extend({ isOn: false, color: 'yellow', age: null, description: Ember.computed('isOn','color',function() { return 'The ' + this.get('color') + ' light is set to ' + this.get('isOn') }), fullDescription: Ember.computed('description','age',function() { return this.get('description') + ' and the age is ' + this.get('age') }), desc: Ember.computed.alias('description'), isOnChanged: Ember.observer('isOn',function() { console.log('isOn value changed') }) }); const bulb = Light.create({age: 22}); bulb.set('isOn',true); //console logs isOn value changed The Ember.observer isOnChanged monitors the isOn property. If any changes occur to this property, isOnChanged is invoked. Computed Properties vs Observers At first glance, it might seem that observers are the same as computed properties. In fact, they are very different. Computed properties can use the get and set methods and can be used in templates. Observers, on the other hand, just monitor property changes. They can neither be used in templates nor be accessed like properties. They also don't return any values. With that said, be careful not to overuse observers. For many instances, a computed property is the most appropriate solution. Also, if required, you can add multiple properties to the observer. Just use the following command: Light.reopen({ isAnythingChanged: Ember.observer('isOn','color',function() { console.log('isOn or color value changed') }) }); const bulb = Light.create({age: 22}); bulb.set('isOn',true); // console logs isOn or color value changed bulb.set('color','blue'); // console logs isOn or color value changed The isAnything observer is invoked whenever the isOn or color properties changes. The observer will fire twice as each property has changed. Synchronous issues with the Light object and observers It's very easy to get observers out of sync. For example, if a property that it observes changes, it will be invoked as expected. After being invoked, it might manipulate a property that hasn't been updated yet. This can cause synchronization issues as everything happens at the same time, as follows: The following example shows this behavior: Light.reopen({ checkIsOn: Ember.observer('isOn', function() { console.log(this.get('fullDescription')); }) }); const bulb = Light.create({age: 22}); bulb.set('isOn', true); When isOn is changed, it's not clear whether fullDescription, a computed property, has been updated yet or not. As observers work synchronously, it's difficult to tell what has been fired and changed. This can lead to unexpected behavior. To counter this, it's best to use the Ember.run.once method. This method is a part of the Ember run loop, which is Ember's way of managing how the code is executed. Reopen the Light object and you can see the following occurring: Light.reopen({ checkIsOn: Ember.observer('isOn','color', function() { Ember.run.once(this,'checkChanged'); }), checkChanged: Ember.observer('description',function() { console.log(this.get('description')); }) }); const bulb = Light.create({age: 22}); bulb.set('isOn', true); bulb.set('color', 'blue'); The checkIsOn observer calls the checkChanged observer using Ember.run.once. This method is only run once per run loop. Normally, checkChanged would be fired twice; however, since it's be called using Ember.run.once, it only outputs once. How it works Ember observers are mixins from the Ember.Observable class. They work by monitoring property changes. When any change occurs, they are triggered. Keep in mind that these are not the same as computed properties and cannot be used in templates or with getters or setters. Summary In this article you learned classes and instances. You also learned computed properties and how they can be used to display data. Resources for Article: Further resources on this subject: Introducing the Ember.JS framework [article] Building Reusable Components [article] Using JavaScript with HTML [article]
Read more
  • 0
  • 0
  • 14632

article-image-chatgpt-as-a-documentation-tool
M.T. White
22 Aug 2023
14 min read
Save for later

ChatGPT as a Documentation Tool

M.T. White
22 Aug 2023
14 min read
It comes as no surprise that most developers do not like writing documentation.  As a result, documentation is often pushed to the side and, more often than not, haphazardly put together.  This is a serious problem since written documentation is the primary way developers communicate how a software system should work and be utilized.  As such, a poorly documented system can render it useless and confine it to the cyber trash heap long before its time.  However, with the rise of new tools like ChatGPT, poor documentation may no longer be a problem.  This article is going to explore how ChatGPT can be leveraged as a code documentation tool and examine the results to see just how well it can document code.What is quality documentation?If you ask five developers what they consider to be good documentation, you’re probably going to get five different answers.  What’s considered good documentation will vary among languages, IDEs, organizational coding standards, and preferences.  As such, for this article, “good” documentation is going to have the following parameters:1.     The documentation will provide context on how a block of code works or how to utilize it. 2.     The documentation will be logical and consistent.  This means that the documentation will be formatted and worded logically to provide context across all code blocks that ChatGPT documents.3.     The autogenerated code documentation will not clutter the code or distract the reader from the task at hand.Again, it is important to remember that there are many different opinions on what quality documentation looks like, and the points above are just high-level metrics to hit.    What is the documentation?Documentation is a very broad term that can mean a lot of things.  It can mean simply the documentation that comes packaged with the codebases like JavaDocs, or it could be a whole ecosystem that ranges from low-level code comments to YouTube videos and Wikipedia articles.  In short, documentation can be anything that helps relay information from one developer to another.  As such, for this article, the term documentation is going to be limited to code comments and high-level documentation like JavaDocs.  Also, most languages have their own best practices for documentation, so for this article, ChatGPT will generate documentation for Java.  Though Java will be the example language, there should only be minor tweaks, if any, to generate documentation for any other language. JavaDocsJavaDocs should provide high-level documentation to relay information about how to use a piece of code to a developer that does not have access to the codebase.  This means that key features like what the methods do, arguments, and return types should be noted.  With that in mind, let’s generate some JavaDocs for the following method,public static void add(int a, int b) {      System.out.println(a + b); }We’re going to generate the JavaDocs for this method using the following prompt:Generate some javadocs for this method public static void add(int a, int b) { System.out.println(a + b); }When the prompt is executed, it will produce the following output:/** * Calculates the sum of two integers and prints the result. * * @param a The first integer operand. * @param b The second integer operand. */ public static void add(int a, int b) {    System.out.println(a + b); }In this case, ChatGPT clearly defined what the arguments are, the datatypes of the arguments, and what the method is meant to do with straightforward prompts.  It is important to note that this method is straightforward, so the next thing to explore is a more complex method.The next method to try to automatically document will be the following:public static double divide(int a, int b) { double div = 0;      if (a > b) {            div = a / b ;      }else {            div = b / a;      }      return div; }For this method, the following prompt will be run. Write only JavaDocs for this method public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }After the prompt is run, we should be met with something similar to the following:/** * Calculates the division of two numbers. * * @param a The numerator. * @param b The denominator. * @return The division result. */These JavaDocs look very similar to the JavaDocs for the simple method.  They provide the arguments, return type, and the purpose of the method.   The one thing that was kind of ambiguous is that ChatGPT attempted to identify the variables as always the numerator or denominator.  From the code, we can see that this isn’t always true; however, if a developer is checking, they should be able to correct it fairly easily. Now, at first glance, this will appear as more trouble than it is worth.  A codebase of significant size is going to have many classes that are comprised of many different methods.  As such, copying and pasting methods one by one is going to be a herculean task.  However, ChatGPT can still be leveraged to speed up high-level code documentation.  Consider the following prompt:Write JavaDocs for these methods public static void add(int a, int b) { System.out.println(a + b); } public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }This prompt will produce the following results:/** * Adds two numbers and prints the result. * * @param a The first number. * @param b The second number. */ /** * Calculates the division of two numbers and returns the result. * * @param a The numerator. * @param b The denominator. * @return The division result. */Again, even with multiple methods, ChatGPT returned JavaDocs that are quality and provided enough information for a developer to use, with the exception of the same hiccup for the division method that was previously explored.  In all, ChatGPT did a decent job with writing JavaDocs.Used in this manner, ChatGPT can greatly enhance the speed and quality of development.  When using ChatGPT, multiple methods can be pasted into ChatGPT, and all the necessary JavaDocs are created.  As anyone who has ever had to write this type of documentation knows, it can be very time-consuming and mistake-prone.  However, with ChatGPT, all one has to do is input the code and let the AI system do the rest.  Ultimately ChatGPT will free up developers for more pressing tasks, and the AI will produce quality JavaDocs without making rudimentary errors like forgetting to include a parameter or return type.  With that, another question is how well ChatGPT can be used to write code comments. Code CommentsIn many cases, it is difficult for a 3rd party to write effective code comments.  To provide the most context to the code, the entity that is writing the code should also write the comments.  As such, the first thing that needs to be explored is how ChatGPT writes code comments.  To do this, consider the following code snippet:public static void add(int a, int b) {      System.out.println(a + b); }This code is very basic; however, in a quality codebase, even a method this simple should be documented.  This method would probably take as much time to document as it would be to write; this means it is a perfect candidate for ChatGPT.  To have ChatGPT document the code snippet runs the following prompt:Write a few comments to provide context for the following method public static void add(int a, int b) { System.out.println(a + b); }After running this prompt, ChatGPT will produce the following:public static void add(int a, int b) {    // This method takes two integer parameters, 'a' and 'b'.    // It adds the values of 'a' and 'b' together and prints the result.    System.out.println(a + b);    // The sum of 'a' and 'b' is calculated and printed to the console using System.out.println().    // This method does not return any value (void). }At first glance, one may notice that ChatGPT generated several comments.  This can be a pro as much as it can be a con since it can bloat the codebase.  With that, one may notice that the output from ChatGPT does have a couple of unnecessary comments that can bloat the code.  For example, there is a comment that says that the method takes in two integers, and another says that the method returns void.  For code comments, those comments are not necessary as they don’t contribute to the overall context of the code.  However, if the prompt is modified to,Write a comment that describes what this method does public static void add(int a, int b) { System.out.println(a + b); }It will produce the following:public static void add(int a, int b) {    // This method calculates the sum of two integers, 'a' and 'b',    // and displays the result by printing it to the console. }This prompt produced two comments that, when combined, form one complete sentence that describes what the method does. Now, this was for a relatively simple method.  As any programmer knows, a method will probably be larger and more complex than the add method.  As such, let’s consider the division method again:public static double divide(int a, int b) {      double div = 0;      if (a > b) {            div = a / b ;      }else {            div = b / a;      }      return div; }This method is a little more complex than the add method.  In this case, this method will return the quotient of a and b if a is greater than b or the quotient of b and a if b is greater than a.  With that, if the following prompt is inputted,Write comments to add context to this method public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }it will produce the following,public static double divide(int a, int b) {    // This method calculates the division of two integers and returns the result.    double div = 0;    // Variable to store the division result.    if (a > b) {        div = a / b;        // If 'a' is greater than 'b', divide 'a' by 'b' and assign the result to 'div'.    } else {        div = b / a;        // If 'b' is greater than or equal to 'a', divide 'b' by 'a' and assign the result to 'div'.    }    return div;    // Return the division result. }In short, ChatGPT will document every line of code again.  The comments do provide code context, but the comments don’t provide anything that can’t simply be read in the code.  Also, if this prompt is modified, ChatGPT, at the time of writing this article, will still produce relatively useless comments.Finally, if a prompt like the following is used:Write only the necessary comments for a developer to use public static double divide(int a, int b) { double div = 0; if (a > b) { div = a / b ; }else { div = b / a; } return div; }it will yield:// Calculates the division of two numbers and returns the result. // Parameters: //   a: The numerator. //   b: The denominator. // Returns: //   The division result.Of all the queries explored thus far, this one arguably yielded the best results.  In this case, the comments are what was generated in the last queries, but the format is much cleaner and easier to modify.  This block could be easily pasted into the codebase, and the unnecessary lines deleted. In all, it took a very specific prompt to produce marginally acceptable code comments.  At best, the code comments that were produced didn’t provide any extra context that could not be deduced from reading the code.  It can also be said that the code comments that were produced bloated the code and, to some, may have made it more confusing.  As such, for code comments, ChatGPT probably isn’t the best tool to use.  At best, a developer will have to remove unnecessary lines of comments and probably have to re-write many of them as well. There is also the issue of having to produce a prompt that is specific enough to generate proper comments. In all, whether a person should use ChatGPT as a code comment generator is up to them.  In theory, the comment produced could be leveraged in places like education, where code examples need to be heavily commented on to provide context to those who may not have a background in the language.  However, in terms of production code, though it will ultimately depend on the organization’s coding standard, ChatGPT will not produce code comments that would be mergeable in many places. Keytake Aways  In terms of codebase comments, ChatGPT is hit-and-miss.  As was seen, the code comments that ChatGPT produced were reminiscent of a college-level developer.  That is, ChatGPT commented on every line of code and only stated the obvious.  Since ChatGPT commented on every line of code, it can be argued that it bloated the codebase to a degree.  However, when a very specific prompt was run, it produced comments similar to what would be found in JavaDocs and what is expected by many organizations.  However, in terms of JavaDocs, ChatGPT shined.  The JavaDocs that ChatGPT produced were all very well written and provided the correct amount of information for a developer to easily digest and apply. As such, a few things can be summarized with what was explored.1.     Queries have to be very specific when it comes to code comments.2.     ChatGPT tends to produce unnecessary code comments that can bloat the codebase. 3.     Depending on the type/quality of code comments, ChatGPT may not be the ideal tool for automatic code documentation.4.     ChatGPT produces documentation akin to JavaDocs better than comments in the codebase.SummaryIn summary, what constitutes quality code documentation is often up to a team.  However, by many standards, ChatGPT tends to produce unnecessary code comments that don’t add much context and can easily bloat the codebase.  However, for higher-level documentation like JavaDocs, ChatGPT is an excellent tool that provides the proper amount of information.  In all, it probably isn’t the best idea to use ChatGPT as a means to generate comments for software written by a human, but it can be used to quickly produce higher-level documentation such as JavaDocs. As was seen, multiple methods can easily be documented in a matter of seconds using ChatGPT.  As such, in terms of productivity, when it comes to higher-level documentation, ChatGPT can be a great productivity tool that could help speed up development. Author BioM.T. White has been programming since the age of 12. His fascination with robotics flourished when he was a child programming microcontrollers such as Arduino. M.T. currently holds an undergraduate degree in mathematics, and a master's degree in software engineering, and is currently working on an MBA in IT project management. M.T. is currently working as a software developer for a major US defense contractor and is an adjunct CIS instructor at ECPI University. His background mostly stems from the automation industry where he programmed PLCs and HMIs for many different types of applications. M.T. has programmed many different brands of PLCs over the years and has developed HMIs using many different tools.Author of the book: Mastering PLC Programming 
Read more
  • 0
  • 0
  • 14624

article-image-developing-basic-site-nodejs-and-express
Packt
17 Feb 2016
21 min read
Save for later

Developing a Basic Site with Node.js and Express

Packt
17 Feb 2016
21 min read
In this article, we will continue with the Express framework. It's one of the most popular frameworks available and is certainly a pioneering one. Express is still widely used and several developers use it as a starting point. (For more resources related to this topic, see here.) Getting acquainted with Express Express (http://expressjs.com/) is a web application framework for Node.js. It is built on top of Connect (http://www.senchalabs.org/connect/), which means that it implements middleware architecture. In the previous chapter, when exploring Node.js, we discovered the benefit of such a design decision: the framework acts as a plugin system. Thus, we can say that Express is suitable for not only simple but also complex applications because of its architecture. We may use only some of the popular types of middleware or add a lot of features and still keep the application modular. In general, most projects in Node.js perform two functions: run a server that listens on a specific port, and process incoming requests. Express is a wrapper for these two functionalities. The following is basic code that runs the server: var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello Worldn'); }).listen(1337, '127.0.0.1'); console.log('Server running at http://127.0.0.1:1337/'); var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello Worldn'); }).listen(1337, '127.0.0.1'); console.log('Server running at http://127.0.0.1:1337/'); This is an example extracted from the official documentation of Node.js. As shown, we use the native module http and run a server on the port 1337. There is also a request handler function, which simply sends the Hello world string to the browser. Now, let's implement the same thing but with the Express framework, using the following code: var express = require('express'); var app = express(); app.get("/", function(req, res, next) { res.send("Hello world"); }).listen(1337); console.log('Server running at http://127.0.0.1:1337/'); It's pretty much the same thing. However, we don't need to specify the response headers or add a new line at the end of the string because the framework does it for us. In addition, we have a bunch of middleware available, which will help us process the requests easily. Express is like a toolbox. We have a lot of tools to do the boring stuff, allowing us to focus on the application's logic and content. That's what Express is built for: saving time for the developer by providing ready-to-use functionalities. Installing Express There are two ways to install Express. We'll will start with the simple one and then proceed to the more advanced technique. The simpler approach generates a template, which we may use to start writing the business logic directly. In some cases, this can save us time. From another viewpoint, if we are developing a custom application, we need to use custom settings. We can also use the boilerplate, which we get with the advanced technique; however, it may not work for us. Using package.json Express is like every other module. It has its own place in the packages register. If we want to use it, we need to add the framework in the package.json file. The ecosystem of Node.js is built on top of the Node Package Manager. It uses the JSON file to find out what we need and installs it in the current directory. So, the content of our package.json file looks like the following code: { "name": "projectname", "description": "description", "version": "0.0.1", "dependencies": { "express": "3.x" } } These are the required fields that we have to add. To be more accurate, we have to say that the mandatory fields are name and version. However, it is always good to add descriptions to our modules, particularly if we want to publish our work in the registry, where such information is extremely important. Otherwise, the other developers will not know what our library is doing. Of course, there are a bunch of other fields, such as contributors, keywords, or development dependencies, but we will stick to limited options so that we can focus on Express. Once we have our package.json file placed in the project's folder, we have to call npm install in the console. By doing so, the package manager will create a node_modules folder and will store Express and its dependencies there. At the end of the command's execution, we will see something like the following screenshot: The first line shows us the installed version, and the proceeding lines are actually modules that Express depends on. Now, we are ready to use Express. If we type require('express'), Node.js will start looking for that library inside the local node_modules directory. Since we are not using absolute paths, this is normal behavior. If we miss running the npm install command, we will be prompted with Error: Cannot find module 'express'. Using a command-line tool There is a command-line instrument called express-generator. Once we run npm install -g express-generator, we will install and use it as every other command in our terminal. If you use the framework inseveral projects, you will notice that some things are repeated. We can even copy and paste them from one application to another, and this is perfectly fine. We may even end up with our own boiler plate and can always start from there. The command-line version of Express does the same thing. It accepts few arguments and based on them, creates a skeleton for use. This can be very handy in some cases and will definitely save some time. Let's have a look at the available arguments: -h, --help: This signifies output usage information. -V, --version: This shows the version of Express. -e, --ejs: This argument adds the EJS template engine support. Normally, we need a library to deal with our templates. Writing pure HTML is not very practical. The default engine is set to JADE. -H, --hogan: This argument is Hogan-enabled (another template engine). -c, --css: If wewant to use the CSS preprocessors, this option lets us use LESS(short forLeaner CSS) or Stylus. The default is plain CSS. -f, --force: This forces Express to operate on a nonempty directory. Let's try to generate an Express application skeleton with LESS as a CSS preprocessor. We use the following line of command: express --css less myapp A new myapp folder is created with the file structure, as seen in the following screenshot: We still need to install the dependencies, so cd myapp && npm install is required. We will skip the explanation of the generated directories for now and will move to the created app.js file. It starts with initializing the module dependencies, as follows: var express = require('express'); var path = require('path'); var favicon = require('static-favicon'); var logger = require('morgan'); var cookieParser = require('cookie-parser'); var bodyParser = require('body-parser'); var routes = require('./routes/index'); var users = require('./routes/users'); var app = express(); Our framework is express, and path is a native Node.js module. The middleware are favicon, logger, cookieParser, and bodyParser. The routes and users are custom-made modules, placed in local for the project folders. Similarly, as in the Model-View-Controller(MVC) pattern, these are the controllers for our application. Immediately after, an app variable is created; this represents the Express library. We use this variable to configure our application. The script continues by setting some key-value pairs. The next code snippet defines the path to our views and the default template engine: app.set('views', path.join(__dirname, 'views')); app.set('view engine', 'jade'); The framework uses the methods set and get to define the internal properties. In fact, we may use these methods to define our own variables. If the value is a Boolean, we can replace set and get with enable and disable. For example, see the following code: app.set('color', 'red'); app.get('color'); // red app.enable('isAvailable'); The next code adds middleware to the framework. Wecan see the code as follows: app.use(favicon()); app.use(logger('dev')); app.use(bodyParser.json()); app.use(bodyParser.urlencoded()); app.use(cookieParser()); app.use(require('less-middleware')({ src: path.join(__dirname, 'public') })); app.use(express.static(path.join(__dirname, 'public'))); The first middleware serves as the favicon of our application. The second is responsible for the output in the console. If we remove it, we will not get information about the incoming requests to our server. The following is a simple output produced by logger: GET / 200 554ms - 170b GET /stylesheets/style.css 200 18ms - 110b The json and urlencoded middleware are related to the data sent along with the request. We need them because they convert the information in an easy-to-use format. There is also a middleware for the cookies. It populates the request object, so we later have access to the required data. The generated app uses LESS as a CSS preprocessor, and we need to configure it by setting the directory containing the .less files. Eventually, we define our static resources, which should be delivered by the server. These are just few lines, but we've configured the whole application. We may remove or replace some of the modules, and the others will continue working. The next code in the file maps two defined routes to two different handlers, as follows: app.use('/', routes); app.use('/users', users); If the user tries to open a missing page, Express still processes the request by forwarding it to the error handler, as follows: app.use(function(req, res, next) { var err = new Error('Not Found'); err.status = 404; next(err); }); The framework suggests two types of error handling:one for the development environment and another for the production server. The difference is that the second one hides the stack trace of the error, which should be visible only for the developers of the application. As we can see in the following code, we are checking the value of the env property and handling the error differently: // development error handler if (app.get('env') === 'development') { app.use(function(err, req, res, next) { res.status(err.status || 500); res.render('error', { message: err.message, error: err }); }); } // production error handler app.use(function(err, req, res, next) { res.status(err.status || 500); res.render('error', { message: err.message, error: {} }); }); At the end, the app.js file exports the created Express instance, as follows: module.exports = app; To run the application, we need to execute node ./bin/www. The code requires app.js and starts the server, which by default listens on port 3000. #!/usr/bin/env node var debug = require('debug')('my-application'); var app = require('../app'); app.set('port', process.env.PORT || 3000); var server = app.listen(app.get('port'), function() { debug('Express server listening on port ' + server.address().port); }); The process.env declaration provides an access to variables defined in the current development environment. If there is no PORT setting, Express uses 3000 as the value. The required debug module uses a similar approach to find out whether it has to show messages to the console. Managing routes The input of our application is the routes. The user visits our page at a specific URL and we have to map this URL to a specific logic. In the context of Express, this can be done easily, as follows: var controller = function(req, res, next) { res.send("response"); } app.get('/example/url', controller); We even have control over the HTTP's method, that is, we are able to catch POST, PUT, or DELETE requests. This is very handy if we want to retain the address path but apply a different logic. For example, see the following code: var getUsers = function(req, res, next) { // ... } var createUser = function(req, res, next) { // ... } app.get('/users', getUsers); app.post('/users', createUser); The path is still the same, /users, but if we make a POST request to that URL, the application will try to create a new user. Otherwise, if the method is GET, it will return a list of all the registered members. There is also a method, app.all, which we can use to handle all the method types at once. We can see this method in the following code snippet: app.all('/', serverHomePage); There is something interesting about the routing in Express. We may pass not just one but many handlers. This means that we can create a chain of functions that correspond to one URL. For example, it we need to know if the user is logged in, there is a module for that. We can add another method that validates the current user and attaches a variable to the request object, as follows: var isUserLogged = function(req, res, next) { req.userLogged = Validator.isCurrentUserLogged(); next(); } var getUser = function(req, res, next) { if(req.userLogged) { res.send("You are logged in. Hello!"); } else { res.send("Please log in first."); } } app.get('/user', isUserLogged, getUser); The Validator class is a class that checks the current user's session. The idea is simple: we add another handler, which acts as an additional middleware. After performing the necessary actions, we call the next function, which passes the flow to the next handler, getUser. Because the request and response objects are the same for all the middlewares, we have access to the userLogged variable. This is what makes Express really flexible. There are a lot of great features available, but they are optional. At the end of this chapter, we will make a simple website that implements the same logic. Handling dynamic URLs and the HTML forms The Express framework also supports dynamic URLs. Let's say we have a separate page for every user in our system. The address to those pages looks like the following code: /user/45/profile Here, 45 is the unique number of the user in our database. It's of course normal to use one route handler for this functionality. We can't really define different functions for every user. The problem can be solved by using the following syntax: var getUser = function(req, res, next) { res.send("Show user with id = " + req.params.id); } app.get('/user/:id/profile', getUser); The route is actually like a regular expression with variables inside. Later, that variable is accessible in the req.params object. We can have more than one variable. Here is a slightly more complex example: var getUser = function(req, res, next) { var userId = req.params.id; var actionToPerform = req.params.action; res.send("User (" + userId + "): " + actionToPerform) } app.get('/user/:id/profile/:action', getUser); If we open http://localhost:3000/user/451/profile/edit, we see User (451): edit as a response. This is how we can get a nice looking, SEO-friendly URL. Of course, sometimes we need to pass data via the GET or POST parameters. We may have a request like http://localhost:3000/user?action=edit. To parse it easily, we need to use the native url module, which has few helper functions to parse URLs: var getUser = function(req, res, next) { var url = require('url'); var url_parts = url.parse(req.url, true); var query = url_parts.query; res.send("User: " + query.action); } app.get('/user', getUser); Once the module parses the given URL, our GET parameters are stored in the .query object. The POST variables are a bit different. We need a new middleware to handle that. Thankfully, Express has one, which is as follows: app.use(express.bodyParser()); var getUser = function(req, res, next) { res.send("User: " + req.body.action); } app.post('/user', getUser); The express.bodyParser() middleware populates the req.body object with the POST data. Of course, we have to change the HTTP method from .get to .post or .all. If we want to read cookies in Express, we may use the cookieParser middleware. Similar to the body parser, it should also be installed and added to the package.json file. The following example sets the middleware and demonstrates its usage: var cookieParser = require('cookie-parser'); app.use(cookieParser('optional secret string')); app.get('/', function(req, res, next){ var prop = req.cookies.propName }); Returning a response Our server accepts requests, does some stuff, and finally, sends the response to the client's browser. This can be HTML, JSON, XML, or binary data, among others. As we know, by default, every middleware in Express accepts two objects, request and response. The response object has methods that we can use to send an answer to the client. Every response should have a proper content type or length. Express simplifies the process by providing functions to set HTTP headers and sending content to the browser. In most cases, we will use the .send method, as follows: res.send("simple text"); When we pass a string, the framework sets the Content-Type header to text/html. It's great to know that if we pass an object or array, the content type is application/json. If we develop an API, the response status code is probably going to be important for us. With Express, we are able to set it like in the following code snippet: res.send(404, 'Sorry, we cannot find that!'); It's even possible to respond with a file from our hard disk. If we don't use the framework, we will need to read the file, set the correct HTTP headers, and send the content. However, Express offers the .sendfile method, which wraps all these operations as follows: res.sendfile(__dirname + "/images/photo.jpg"); Again, the content type is set automatically; this time it is based on the filename's extension. When building websites or applications with a user interface, we normally need to serve an HTML. Sure, we can write it manually in JavaScript, but it's good practice to use a template engine. This means we save everything in external files and the engine reads the markup from there. It populates them with some data and, at the end, provides ready-to-show content. In Express, the whole process is summarized in one method, .render. However, to work properly, we have to instruct the framework regarding which template engine to use. We already talked about this in the beginning of this chapter. The following two lines of code, set the path to our views and the template engine: app.set('views', path.join(__dirname, 'views')); app.set('view engine', 'jade'); Let's say we have the following template ( /views/index.jade ): h1= title p Welcome to #{title} Express provides a method to serve templates. It accepts the path to the template, the data to be applied, and a callback. To render the previous template, we should use the following code: res.render("index", {title: "Page title here"}); The HTML produced looks as follows: <h1>Page title here</h1><p>Welcome to Page title here</p> If we pass a third parameter, function, we will have access to the generated HTML. However, it will not be sent as a response to the browser. The example-logging system We've seen the main features of Express. Now let's build something real. The next few pages present a simple website where users can read only if they are logged in. Let's start and set up the application. We are going to use Express' command-line instrument. It should be installed using npm install -g express-generator. We create a new folder for the example, navigate to it via the terminal, and execute express --css less site. A new directory, site, will be created. If we go there and run npm install, Express will download all the required dependencies. As we saw earlier, by default, we have two routes and two controllers. To simplify the example, we will use only the first one: app.use('/', routes). Let's change the views/index.jade file content to the following HTML code: doctype html html head title= title link(rel='stylesheet', href='/stylesheets/style.css') body h1= title hr p That's a simple application using Express. Now, if we run node ./bin/www and open http://127.0.0.1:3000, we will see the page. Jade uses indentation to parse our template. So, we should not mix tabs and spaces. Otherwise, we will get an error. Next, we need to protect our content. We check whether the current user has a session created; if not, a login form is shown. It's the perfect time to create a new middleware. To use sessions in Express, install an additional module: express-session. We need to open our package.json file and add the following line of code: "express-session": "~1.0.0" Once we do that, a quick run of npm install will bring the module to our application. All we have to do is use it. The following code goes to app.js: var session = require('express-session'); app.use(session({ secret: 'app', cookie: { maxAge: 60000 }})); var verifyUser = function(req, res, next) { if(req.session.loggedIn) { next(); } else { res.send("show login form"); } } app.use('/', verifyUser, routes); Note that we changed the original app.use('/', routes) line. The session middleware is initialized and added to Express. The verifyUser function is called before the page rendering. It uses the req.session object, and checks whether there is a loggedIn variable defined and if its value is true. If we run the script again, we will see that the show login form text is shown for every request. It's like this because no code sets the session exactly the way we want it. We need a form where users can type their username and password. We will process the result of the form and if the credentials are correct, the loggedIn variable will be set to true. Let's create a new Jade template, /views/login.jade: doctype html html head title= title link(rel='stylesheet', href='/stylesheets/style.css') body h1= title hr form(method='post') label Username: br input(type='text', name='username') br label Password: br input(type='password', name='password') br input(type='submit') Instead of sending just a text with res.send("show login form"); we should render the new template, as follows: res.render("login", {title: "Please log in."}); We choose POST as the method for the form. So, we need to add the middleware that populates the req.body object with the user's data, as follows: app.use(bodyParser()); Process the submitted username and password as follows: var verifyUser = function(req, res, next) { if(req.session.loggedIn) { next(); } else { var username = "admin", password = "admin"; if(req.body.username === username && req.body.password === password) { req.session.loggedIn = true; res.redirect('/'); } else { res.render("login", {title: "Please log in."}); } } } The valid credentials are set to admin/admin. In a real application, we may need to access a database or get this information from another place. It's not really a good idea to place the username and password in the code; however, for our little experiment, it is fine. The previous code checks whether the passed data matches our predefined values. If everything is correct, it sets the session, after which the user is forwarded to the home page. Once you log in, you should be able to log out. Let's add a link for that just after the content on the index page (views/index.jade ): a(href='/logout') logout Once users clicks on this link, they will be forward to a new page. We just need to create a handler for the new route, remove the session, and forward them to the index page where the login form is reflected. Here is what our logging out handler looks like: // in app.js var logout = function(req, res, next) { req.session.loggedIn = false; res.redirect('/'); } app.all('/logout', logout); Setting loggedIn to false is enough to make the session invalid. The redirect sends users to the same content page they came from. However, this time, the content is hidden and the login form pops up. Summary In this article, we learned about one of most widely used Node.js frameworks, Express. We discussed its fundamentals, how to set it up, and its main characteristics. The middleware architecture, which we mentioned in the previous chapter, is the base of the library and gives us the power to write complex but, at the same time, flexible applications. The example we used was a simple one. We required a valid session to provide page access. However, it illustrates the usage of the body parser middleware and the process of registering the new routes. We also updated the Jade templates and saw the results in the browser. For more information on Node.js Refer to the following URLs: https://www.packtpub.com/web-development/instant-nodejs-starter-instant https://www.packtpub.com/web-development/learning-nodejs-net-developers https://www.packtpub.com/web-development/nodejs-essentials Resources for Article: Further resources on this subject: Writing a Blog Application with Node.js and AngularJS [article] Testing in Node and Hapi [article] Learning Node.js for Mobile Application Development [article]
Read more
  • 0
  • 0
  • 14618
article-image-ai-distilled-28-unveiling-innovations-reshaping-our-world
Merlyn Shelley
11 Dec 2023
13 min read
Save for later

AI_Distilled #28: Unveiling Innovations Reshaping Our World

Merlyn Shelley
11 Dec 2023
13 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!👋 Hello ,“Generative AI has the potential to change the world in ways that we can’t even imagine. It has the power to create new ideas, products, and services that will make our lives easier, more productive, and more creative. It also has the potential to solve some of the world’s biggest problems, such as climate change, poverty, and disease.” -Bill Gates, Microsoft Co-Founder Microsoft Bing’s new Deep Search functionality is a case in point — Bing will now create AI prompts itself to provide detailed insights to user queries in ways traditional search engines can’t even match. Who could have thought LLMs would progress so much they would eventually prompt themselves? Even Runway ML is onto something big with its groundbreaking technology that creates realistic AI generated videos that will find their way to Hollywood. Welcome back to a new issue of AI Distilled - your one-stop destination for all things AI, ML, NLP, and Gen AI. Let’s get started with the latest news and developments across the AI sector:  Elon Musk's xAI Initiates $1 Billion Funding Drive in AI Race Bing’s New Deep Search Expands Queries AI Takes Center Stage in 2023 Word of the Year Lists OpenAI Announces Delay in GPT Store Launch to Next Year ChatGPT Celebrates First Anniversary with 110M Installs and $30M Revenue Milestone Runway ML and Getty Images Collaborate on AI Video Models for Hollywood and Advertising We’ve also curated the latest GPT and LLM resources, tutorials, and secret knowledge: Unlocking AI Magic: A Primer on 7 Essential Libraries for Developers Efficient LLM Fine-Tuning with QLoRA on a Laptop Rapid Deployment of Large Open Source LLMs with Runpod and vLLM’s OpenAI Endpoint Understanding Strategies to Enhance Retrieval-Augmented Generation (RAG) Pipeline Performance Understanding and Mitigating Biases and Toxicity in LLMs Finally, don’t forget to check-out our hands-on tips and strategies from the AI community for you to use on your own projects: A Step-by-Step Guide to Streamlining LLM Data Processing for Efficient Pipelines Fine-Tuning Mistral Instruct 7B on the MedMCQA Dataset Using QLoRA Accelerating Large-Scale Training: A Comprehensive Guide to Amazon SageMaker Data Parallel Library Enhancing LoRA-Based Inference Speed: A Guide to Efficient LoRA Decomposition Looking for some inspiration? Here are some GitHub repositories to get your projects going! tacju/maxtron Tanuki/tanuki.py roboflow/multimodal-maestro 03axdov/muskie Also, don't forget to check our expert insights column, which covers the interesting concepts of NLP from the book 'The Handbook of NLP with Gensim'. It's a must-read!    Stay curious and gear up for an intellectually enriching experience! 📥 Feedback on the Weekly EditionQuick question: How can we foster effective collaboration between humans and AI systems, ensuring that AI complements human skills and enhances productivity without causing job displacement or widening societal gaps?Share your valued opinions discreetly! Your insights could shine in our next issue for the 39K-strong AI community. Join the conversation! 🗨️✨ As a big thanks, get our bestselling "Interactive Data Visualization with Python - Second Edition" in PDF.  Let's make AI_Distilled even more awesome! 🚀 Jump on in! Share your thoughts and opinions here! Writer’s Credit: Special shout-out to Vidhu Jain for their valuable contribution to this week’s newsletter content!  Cheers,  Merlyn Shelley  Editor-in-Chief, Packt  SignUp | Advertise | Archives⚡ TechWave: AI/GPT News & Analysis🏐 Elon Musk's xAI Initiates $1 Billion Funding Drive in AI Race: xAI is on a quest to secure $1 billion in equity, aiming to stay competitive with tech giants like OpenAI, Microsoft, and Google in the dynamic AI landscape. Already amassing $135 million from investors, xAI's total funding goal is disclosed in a filing with the US Securities and Exchange Commission.  🏐 AI Alliance Launched by Tech Giants IBM and Meta: IBM and Meta have formed a new "AI Alliance" with over 50 partners to promote open and responsible AI development. Members include Dell, Intel, CERN, NASA and Sony. The alliance envisions fostering an open AI community for researchers and developers and can help members make progress if they openly share models or not. 🏐 Bing’s New Deep Search Expands Queries: Microsoft is testing a new Bing feature called Deep Search that uses GPT-4 to expand search queries before providing results. Deep Search displays the expanded topics in a panel for users to select the one that best fits what they want to know. It then tailors the search results to that description. Microsoft says the feature can take up to 30 seconds due to the AI generation. 🏐 AI Takes Center Stage in 2023 Word of the Year Lists: In 2023, AI dominates tech, influencing "word of the year" choices. Cambridge picks "hallucinate" for AI's tendency to invent information; Merriam-Webster chooses "authentic" to address AI's impact on reality. Oxford recognizes "prompt" for its evolved role in instructing generative AI, reflecting society's increased integration of AI into everyday language and culture. 🏐 OpenAI Announces Delay in GPT Store Launch to Next Year: OpenAI delays the GPT store release until next year, citing unexpected challenges and postponing the initial December launch plan. Despite recent challenges, including CEO changes and employee unrest, development continues, and updates for ChatGPT are expected. The GPT store aims to be a marketplace for users to sell and share custom GPTs, with creators compensated based on usage. 🏐 ChatGPT Celebrates First Anniversary with 110M Installs and $30M Revenue Milestone: ChatGPT's mobile apps, launched in May 2023 on iOS and later on Android, have exceeded 110 million installs, yielding nearly $30 million in revenue. The success is fueled by the ChatGPT Plus subscription, offering perks. Despite competition, downloads surge, with Android hitting 18 million in a week. The company expects continued growth by year-end 2023. 🏐 Runway ML and Getty Images Collaborate on AI Video Models for Hollywood and Advertising: NYC video AI startup Runway ML, backed by Google and NVIDIA, announces a partnership with Getty Images for the Runway <> Getty Images Model (RGM), a generative AI video model. Targeting Hollywood, advertising, media, and broadcasting, it enables customized content workflows for Runway enterprise customers. 🔮 Expert Insights from Packt Community The Handbook of NLP with Gensim - By Chris Kuo NLU + NLG = NLP NLP is an umbrella term that covers natural language understanding (NLU) and NLG. We’ll go through both in the next sections. NLU Many languages, such as English, German, and Chinese, have been developing for hundreds of years and continue to evolve. Humans can use languages artfully in various social contexts. Now, we are asking a computer to understand human language. What’s very rudimentary to us may not be so apparent to a computer. Linguists have contributed much to the development of computers’ understanding in terms of syntax, semantics, phonology, morphology, and pragmatics. NLU focuses on understanding the meaning of human language. It extracts text or speech input and then analyzes the syntax, semantics, phonology, morphology, and pragmatics in the language. Let’s briefly go over each one: Syntax: This is about the study of how words are arranged to form phrases and clauses, as well as the use of punctuation, order of words, and sentences. Semantics: This is about the possible meanings of a sentence based on the interactions between words in the sentence. It is concerned with the interpretation of language, rather than its form or structure. For example, the word “table” as a noun can refer to “a piece of furniture having a smooth flat top that is usually supported by one or more vertical legs” or a data frame in a computer language. NLU can understand the two meanings of a word in such jokes through a technique called word embedding.  Phonology: This is about the study of the sound system of a language, including the sounds of speech (phonemes), how they are combined to form words (morphology), and how they are organized into larger units such as syllables and stress patterns. For example, the sounds represented by the letters “p” and “b” in English are distinct phonemes. A phoneme is the smallest unit of sound in a language that can change the meaning of a word. Consider the words “pat” and “bat.” The only difference between these two words is the initial sound, but their meanings are different. Morphology: This is the study of the structure of words, including the way in which they are formed from smaller units of meaning called morphemes. It originally comes from “morph,” the shape or form, and “ology,” the study of something. Morphology is important because it helps us understand how words are formed and how they relate to each other. It also helps us understand how words change over time and how they are related to other words in a language. For example, the word “unkindness” consists of three separate morphemes: the prefix “un-,” the root “kind,” and the suffix “-ness.” Pragmatics: This is the study of how language is used in a social context. Pragmatics is important because it helps us understand how language works in real-world situations, and how language can be used to convey meaning and achieve specific purposes. For example, if you offer to buy your friend a McDonald’s burger, a large fries, and a large drink, your friend may reply "no" because he is concerned about becoming fat. Your friend may simply mean the burger meal is high in calories, but the conversation can also imply he may be fat in a social context. Now, let’s understand NLG. NLG While NLU is concerned with reading for a computer to comprehend, NLG is about writing for a computer to write. The term generation in NLG refers to an NLP model generating meaningful words or even articles. Today, when you compose an email or type a sentence in an app, it presents possible words to complete your sentence or performs automatic correction. These are applications of NLG.  This content is from the book The Handbook of NLP with Gensim - By Chris Kuo (Oct 2023). Start reading a free chapter or access the entire Packt digital library free for 7 days by signing up now. To learn more, click on the button below. Read through the Chapter 1 unlocked here...  🌟 Secret Knowledge: AI/LLM Resources🏀 Unlocking AI Magic: A Primer on 7 Essential Libraries for Developers: Discover seven cutting-edge libraries to enhance development projects with advanced AI features. From CopilotTextarea for AI-driven writing in React apps to PrivateGPT for secure, locally processed document interactions, explore tools that elevate your projects and impress users. 🏀 Efficient LLM Fine-Tuning with QLoRA on a Laptop: Explore QLoRA, an efficient memory-saving method for fine-tuning large language models on ordinary CPUs. The QLoRA API supports NF4, FP4, INT4, and INT8 data types for quantization, utilizing methods like LoRA and gradient checkpointing to significantly reduce memory requirements. Learn to implement QLoRA on CPUs, leveraging Intel Extension for Transformers, with experiments showcasing its efficiency on consumer-level CPUs. 🏀 Rapid Deployment of Large Open Source LLMs with Runpod and vLLM’s OpenAI Endpoint: Learn to swiftly deploy open-source LLMs into applications with a tutorial, featuring the Llama-2 70B model and AutoGen framework. Utilize tools like Runpod and vLLM for computational resources and API endpoint creation, with a step-by-step guide and the option for non-gated models like Falcon-40B. 🏀 Understanding Strategies to Enhance Retrieval-Augmented Generation (RAG) Pipeline Performance: Learn optimization techniques for RAG applications by focusing on hyperparameters, tuning strategies, data ingestion, and pipeline preparation. Explore improvements in inferencing through query transformations, retrieval parameters, advanced strategies, re-ranking models, LLMs, and prompt engineering for enhanced retrieval and generation. 🏀 Understanding and Mitigating Biases and Toxicity in LLMs: Explore the impact of ethical guidelines on Large Language Model (LLM) development, examining measures adopted by companies like OpenAI and Google to address biases and toxicity. Research covers content generation, jailbreaking, and biases in diverse domains, revealing complexities and challenges in ensuring ethical LLMs.  🔛 Masterclass: AI/LLM Tutorials🎯 A Step-by-Step Guide to Streamlining LLM Data Processing for Efficient Pipelines: Learn to optimize the development loop for your LLM-powered recommendation system by addressing slow processing times in data pipelines. The solution involves implementing a Pipeline class to save inputs/outputs, enabling efficient error debugging. Enhance developer experience with individual pipeline stages as functions and consider future optimizations like error classes and concurrency. 🎯 Fine-Tuning Mistral Instruct 7B on the MedMCQA Dataset Using QLoRA: Explore fine-tuning Mistral Instruct 7B, an open-source LLM, for medical entrance exam questions using the MedMCQA dataset. Utilize Google Colab, GPTQ version, and LoRA technique for memory efficiency. The tutorial covers data loading, prompt creation, configuration, training setup, code snippets, and performance evaluation, offering a foundation for experimentation and enhancement. 🎯 Accelerating Large-Scale Training: A Comprehensive Guide to Amazon SageMaker Data Parallel Library: This guide details ways to boost Large Language Model (LLM) training speed with Amazon SageMaker's SMDDP. It addresses challenges in distributed training, emphasizing SMDDP's optimized AllGather for GPU communication bottleneck, exploring techniques like EFA network usage, GDRCopy coordination, and reduced GPU streaming multiprocessors for improved efficiency and cost-effectiveness on Amazon SageMaker. 🎯 Enhancing LoRA-Based Inference Speed: A Guide to Efficient LoRA Decomposition: The article highlights achieving three times faster inference for public LoRAs using the Diffusers library. It introduces LoRA, a parameter-efficient fine-tuning technique, detailing its decomposition process and benefits, including quick transitions and reduced warm-up and response times in the Inference API.  🚀 HackHub: Trending AI Tools⚽ tacju/maxtron: Unified meta-architecture for video segmentation, enhancing clip-level segmenters with within-clip and cross-clip tracking modules. ⚽ Tanuki/tanuki.py: Simplifies the creation of apps powered by LLMs in Python by seamlessly integrating well-typed, reliable, and stateless LLM-powered functions into applications. ⚽ roboflow/multimodal-maestro: Empowers developers with enhanced control over large multimodal models, enabling the achievement of diverse outputs through effective prompting tactics. ⚽ 03axdov/muskie: Python-based ML library that simplifies the process of dataset creation and model utilization, aiming to reduce code complexity. 
Read more
  • 0
  • 0
  • 14618

article-image-asking-if-developers-can-be-great-entrepreneurs-is-like-asking-if-moms-can-excel-at-work-and-motherhood
Aarthi Kumaraswamy
05 Jun 2018
13 min read
Save for later

Asking if good developers can be great entrepreneurs is like asking if moms can excel at work and motherhood

Aarthi Kumaraswamy
05 Jun 2018
13 min read
You heard me right. Asking, ‘can developers succeed at entrepreneurship?’ is like asking ‘if women should choose between work and having children’. ‘Can you have a successful career and be a good mom?’ is a question that many well-meaning acquaintances still ask me. You see I am a new mom, (not a very new one, my son is 2.5 years old now). I’m also the managing editor of this site since its inception last year when I rejoined work after a maternity break. In some ways, the Packt Hub feels like running a startup too. :) Now how did we even get to this question? It all started with the results of this year's skill up survey. Every year we conduct a survey amongst developers to know the pulse of the industry and to understand their needs, aspirations, and apprehensions better so that we can help them better to skill up. One of the questions we asked them this year was: ‘Where do you see yourself in the next 5 years?’ To this, an overwhelming one third responded stating they want to start a business. Surveys conducted by leading consultancies, time and again, show that only 2 or 3 in 10 startups survive. Naturally, a question that kept cropping up in our editorial discussions after going through the above response was: Can developers also succeed as entrepreneurs? To answer this, first let’s go back to the question: Can you have a successful career and be a good mom? The short answer is, Yes, it is possible to be both, but it will be hard work. The long answer is, This path is not for everyone. As a working mom, you need to work twice as hard, befriend uncertainty and nurture a steady support system that you trust both at work and home to truly flourish. At times, when you see your peers move ahead in their careers or watch stay-at-home moms with their kids, you might feel envy, inadequacy or other unsavory emotions in between. You need superhuman levels of mental, emotional and physical stamina to be the best version of yourself to move past such times with grace, and to truly appreciate the big picture: you have a job you love and a kid who loves you. But what has my experience as a working mom got to do anything with developers who want to start their own business? You’d be surprised to see the similarities. Starting a business is, in many ways, like having a baby. There is a long incubation period, then the painful launch into the world followed by sleepless nights of watching over the business so it doesn’t choke on itself while you were catching a nap. And you are doing this for the first time too, even if you have people giving you advice. Then there are those sustenance issues to look at and getting the right people in place to help the startup learn to turn over, sit up, stand up, crawl and then finally run and jump around till it makes you go mad with joy and apprehension at the same time thinking about what’s next in store now. How do I scale my business? Does my business even need me? To me, being a successful developer, writer, editor or any other profession, for that matter, is about being good at what you do (write code, write stories, spot raw talent, and bring out the best in others etc) and enjoying doing it immensely. It requires discipline, skill, expertise, and will. To see if the similarities continue, let’s try rewriting my statement for a developer turned entrepreneur. Can you be a good developer and a great entrepreneur? This path is not for everyone. As a developer-turned-entrepreneur, you need to work twice as hard as your friends who have a full-time job, to make it through the day wearing many hats and putting out workplace fires that have got nothing to do with your product development. You need a steady support system both at work and home to truly flourish. At times, when you see others move ahead in their careers or listen to entrepreneurs who have sold their business to larger companies or just got VC funded, you might feel envy, selfishness, inadequacy or any other emotion in between. You need superhuman levels of maturity to move past such times, and to truly appreciate the big picture: you built something incredible and now you are changing the world, even if it is just one customer at a time with your business. Now that we sail on the same boat, here are the 5 things I learned over the last year as a working mom that I hope you as a developer-entrepreneur will find useful. I’d love to hear your take on them in the comments below. [divider style="normal" top="20" bottom="20"] #1. Become a productivity master. Compartmentalize your roles and responsibilities into chunks spread across the day. Ruthlessly edit out clutter from your life. [divider style="normal" top="20" bottom="20"] Your life changed forever when your child (business) was born. What worked for you as a developer may not work as an entrepreneur. The sooner you accept this, the faster you will see the differences and adapt accordingly. For example, I used to work quite late into the night and wake up late in the morning before my son was born. I also used to binge watch TV shows during weekends. Now I wake up as early as 4.30 AM so that I can finish off the household chores for the day and get my son ready for play school by 9 AM. I don’t think about work at all at this time. I must accommodate my son during lunch break and have a super short lunch myself. But apart from that I am solely focused on the work at hand during office hours and don’t think about anything else. My day ends at 11 PM. Now I am more choosy about what I watch as I have very less time for leisure. I instead prefer to ‘do’ things like learning something new, spending time bonding with my son, or even catching up on sleep, on weekends. This spring-cleaning and compartmentalization took a while to get into habit, but it is worth the effort. Truth be told, I still occasionally fallback to binging, like I am doing right now with this article, writing it at 12 AM on a Saturday morning because I’m in a flow. As a developer-entrepreneur, you might be tempted to spend most of your day doing what you love, i.e., developing/creating something because it is a familiar territory and you excel at that. But doing so at the cost of your business will cost you your business, sooner than you think. Resist the urge and instead, organize your day and week such that you spend adequate time on key aspects of your business including product development. Make a note of everything you do for a few days and then decide what’s not worth doing and what else can you do instead in its place. Have room only for things that matter to you which enrich your business goals and quality of life. [divider style="normal" top="20" bottom="20"] #2. Don’t aim to create the best, aim for good enough. Ship the minimum viable product (MVP). [divider style="normal" top="20" bottom="20"] All new parents want the best for their kids from the diaper they use to the food they eat and the toys they play with. They can get carried away buying stuff that they think their child needs only to have a storeroom full of unused baby items. What I’ve realized is that babies actually need very less stuff. They just need basic food, regular baths, clean diapers (you could even get away without those if you are up for the challenge!), sleep, play and lots of love (read mommy and daddy time, not gifts). This is also true for developers who’ve just started up. They want to build a unique product that the world has never seen before and they want to ship the perfect version with great features and excellent customer reviews. But the truth is, your first product is, in fact, a prototype built on your assumption of what your customer wants. For all you know, you may be wrong about not just your customer’s needs but even who your customer is. This is why a proper market study is key to product development. Even then, the goal should be to identify the key features that will make your product viable. Ship your MVP, seek quick feedback, iterate and build a better product in the next version. This way you haven’t unnecessarily sunk upfront costs, you’re ahead of your competitors and are better at estimating customer needs as well. Simplicity is the highest form of sophistication. You need to find just the right elements to keep in your product and your business model. As Michelangelo would put it, “chip away the rest”. [divider style="normal" top="20" bottom="20"] #3. There is no shortcut to success. Don’t compromise on quality or your values. [divider style="normal" top="20" bottom="20"] The advice to ship a good enough product is not a permission to cut corners. Since their very first day, babies watch and learn from us. They are keen observers and great mimics. They will feel, talk and do what we feel, say and do, even if they don’t understand any of the words or gestures. I think they do understand emotions. One more reason for us to be better role models for our children. The same is true in a startup. The culture mimics the leader and it quickly sets in across the organization. As a developer, you may have worked long hours, even into the night to get that piece of code working and felt a huge rush from the accomplishment. But as an entrepreneur remember that you are being watched and emulated by those who work for you. You are what they want to be when they ‘grow up’. Do you want a crash and burn culture at your startup? This is why it is crucial that you clarify your business’s goals, purpose, and values to everyone in your company, even if it just has one other person working there. It is even more important that you practice what you preach. Success is an outcome of right actions taken via the right habits cultivated. [divider style="normal" top="20" bottom="20"] #4. You can’t do everything yourself and you can’t be everywhere. You need to trust others to do their job and appreciate them for a job well done. This also means you hire the right people first. [divider style="normal" top="20" bottom="20"] It takes a village to raise a child, they say. And it is very true, especially in families where both parents work. It would’ve been impossible for me to give my 100% at work, if I had to keep checking in on how my son is doing with his grandparents or if I refused the support my husband offers sharing household chores. Just because you are an exceptional developer, you can’t keep micromanaging product development at your startup. As a founder, you have a larger duty towards your entire business and customers. While it is good to check how your product is developing, your pure developer days are behind. Find people better than you at this job and surround yourself with people who are good at what they do, and share your values for key functions of your startup. Products succeed because of developers, businesses succeed because their leader knew when to get out of the way and when to step in. [divider style="normal" top="20" bottom="20"] #5. Customer first, product next, ego last. [divider style="normal" top="20" bottom="20"] This has been the toughest lesson so far. It looks deceptively simple in theory but hard to practice in everyday life. As developers and engineers, we are primed to find solutions. We also see things in binaries: success and failure, right and wrong, good and bad. This is a great quality to possess to build original products. However, it is also the Achilles heel for the developer turned entrepreneur. In business, things are seldom in black and white. Putting product first can be detrimental to knowing your customers. For example, you build a great product, but the customer is not using it as you intended it to be used. Do you train your customers to correct their ways or do you unearth the underlying triggers that contribute to that customer behavior and alter your product’s vision? The answer is, ‘it depends’. And your job as an entrepreneur is to enable your team to find the factors that influence your decision on the subject; it is not to find the answer yourself or make a decision based on your experience. You need to listen more than you talk to truly put your customer first. To do that you need to acknowledge that you don’t know everything. Put your ego last. Make it easy for customers to share their views with you directly. Acknowledge that your product/service exists to serve your customers’ needs. It needs to evolve with your customer. Find yourself a good mentor or two who you respect and want to emulate. They will be your sounding board and the light at the end of the tunnel during your darkest hours (you will have many of those, I guarantee). Be grateful for your support network of family, friends, and colleagues and make sure to let them know how much you appreciate them for playing their part in your success. If you have the right partner to start the business, jump in with both feet. Most tech startups have a higher likelihood of succeeding when they have more than one founder. Probably because the demands of tech and business are better balanced on more than one pair of shoulders. [dropcap]H[/dropcap]ow we frame our questions is a big part of the problem. Reframing them makes us see different perspectives thereby changing our mindsets. Instead of asking ‘can working moms have it all?’, what if we asked ‘what can organizations do to help working moms achieve work-life balance?’, ‘how do women cope with competing demands at work and home?’ Instead of asking ‘can developers be great entrepreneurs?’ better questions to ask are ‘what can developers do to start a successful business?’, ‘what can we learn from those who have built successful companies?’ Keep an open mind; the best ideas may come from the unlikeliest sources! 1 in 3 developers wants to be an entrepreneur. What does it take to make a successful transition? Developers think managers don’t know enough about technology. And that’s hurting business. 96% of developers believe developing soft skills is important  
Read more
  • 0
  • 0
  • 14615
Modal Close icon
Modal Close icon